KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r345643M: Thu Mar 28 16:37:31 CET 2019 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 8.0.0 (branches/release_80 356034) (based on LLVM 8.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8252743680 (7870 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: unblocking device. ioapic0 irqs 0-23 on motherboard Launching APs: 3 1 2 Timecounter "TSC" frequency 1800014712 Hz quality 800 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0x14d2320, 0) error 19 kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 vgapci0: Boot video device hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahci0: quirks=0x7000 ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1 on ohci1 usbus1: 12Mbps Full Speed USB v1.0 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2 on ohci2 usbus2: 12Mbps Full Speed USB v1.0 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3 on ohci3 usbus3: 12Mbps Full Speed USB v1.0 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4 on ohci4 usbus4: 12Mbps Full Speed USB v1.0 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5 on ehci0 usbus5: 480Mbps High Speed USB v2.0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pci3: at device 6.0 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. hwpstate0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 3 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 20,22,21,23 and 24,26 on hdaa1 pcm2: at nid 27 and 25 on hdaa1 pcm3: at nid 30 on hdaa1 ugen3.1: at usbus3 ugen4.1: at usbus4 uhub0: ugen2.1: at usbus2 ugen5.1: at usbus5 on usbus3 uhub2: on usbus5 uhub3: on usbus4 uhub1: on usbus2 ugen0.1: at usbus0 ugen1.1: at usbus1 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 uhub4: Expensive timeout(9) function: 0xa3a470(0x21f7c000) 0.008957283 s on usbus0 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout uhub5: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted on usbus1 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) cd0 at ata0 bus 0 scbus4 target 1 lun 0 cd0: <_NEC DVD_RW ND-4550A 1.06> Removable CD-ROM SCSI device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 610462 free (2638 frags, 75978 blocks, 0.3% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 779801 free (11089 frags, 96089 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 27463069 free (36549 frags, 3428315 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1528671 free (284743 frags, 155491 blocks, 2.8% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2904200 free (155160 frags, 343630 blocks, 0.6% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. uhub2: 10 ports with 10 removable, self powered Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0x1246366c bufwait (bufwait) @ kern/vfs_bio.c:3906 2nd 0x3002ac00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfff79d at _sx_xlock+0x4d #3 0x12d3dc5 at ufsdirhash_add+0x45 #4 0x12d662e at ufs_direnter+0x3fe #5 0x12de0bc at ufs_mkdir+0x78c #6 0x15907ad at VOP_MKDIR_APV+0x6d #7 0x10bf6c3 at kern_mkdirat+0x173 #8 0x10bf549 at sys_mkdir+0x19 #9 0x15616b9 at syscall+0x2d9 #10 0xffc033b7 at _end+0xfdc0c84f Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Mar 30 19:25:34 x4 smartd[634]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Mar 30 19:25:34 x4 smartd[634]: see the following Seagate web pages: Mar 30 19:25:34 x4 smartd[634]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Mar 30 19:25:34 x4 smartd[634]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Sat Mar 30 19:25 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Mar 30 19:25:48 x4 su[807]: pho to root on /dev/pts/0 lock order reversal: 1st 0x32121a28 msdosfs (msdosfs) @ kern/vfs_subr.c:3112 2nd 0x32a9f8bc devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:854 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcffb5 at lockmgr_lock_fast_path+0x125 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0xedd923 at msdosfs_fsync+0x43 #7 0x1590219 at VOP_FSYNC_APV+0x69 #8 0x10b291b at vflush+0x20b #9 0xedaf64 at msdosfs_mount+0xa4 #10 0x10a9c2e at vfs_domount+0x60e #11 0x10a910e at vfs_donmount+0x79e #12 0x10a893d at sys_nmount+0x5d #13 0x15616b9 at syscall+0x2d9 #14 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 lock order reversal: 1st 0x30681e6c ufs (ufs) @ kern/vfs_mount.c:1353 2nd 0x32a9f8bc devfs (devfs) @ fs/msdosfs/msdosfs_vfsops.c:937 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcffb5 at lockmgr_lock_fast_path+0x125 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0xedc473 at msdosfs_sync+0x173 #7 0x10aaed5 at dounmount+0x425 #8 0x10aa9bb at sys_unmount+0x26b #9 0x15616b9 at syscall+0x2d9 #10 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 20190330 19:26:42 all (1/44): buildworld2.sh 20190330 19:26:48 all (2/44): cmp.sh mount leak: 299/397 20190330 19:32:31 all (3/44): core3.sh lock order reversal: 1st 0x33999478 ufs (ufs) @ kern/vfs_subr.c:2598 2nd 0x124cdb88 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:283 3rd 0x339805e4 ufs (ufs) @ kern/vfs_subr.c:2598 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfd03d8 at lockmgr_xlock_hard+0x48 #3 0xfd0b46 at __lockmgr_args+0x376 #4 0x12cedcf at ffs_lock+0x6f #5 0x1591158 at VOP_LOCK1_APV+0x58 #6 0x10c2858 at _vn_lock+0x58 #7 0x10b1b8b at vget+0x7b #8 0x10a5013 at vfs_hash_get+0xa3 #9 0x12ca8c7 at ffs_vgetf+0x27 #10 0x12c0a3f at softdep_sync_buf+0x2cf #11 0x12cfaf1 at ffs_syncvnode+0x231 #12 0x12cd37c at ffs_sync+0x1dc #13 0x10c3c62 at vfs_write_suspend+0x122 #14 0x10c3e0c at vfs_write_suspend_umnt+0x3c #15 0x12ccf5c at ffs_unmount+0x3ec #16 0x10aaec5 at dounmount+0x415 #17 0x10aa9bb at sys_unmount+0x26b Mar 30 19:32:36 x4 kernel: Failed to write core file for process core3 (error 5) Mar 30 19:33:07 x4 syslogd: last message repeated 9592 times Mar 30 19:35:08 x4 syslogd: last message repeated 38898 times Mar 30 19:42:35 x4 syslogd: last message repeated 139889 times mount leak: 24634/25031 GEOM leak: 1165/1418 20190330 19:52:38 all (4/44): core5.sh freework leak: 13/26. 20190330 20:02:46 all (5/44): credleak.sh mount leak: 147/25178 20190330 20:04:01 all (6/44): datamove4.sh mount leak: 1/25179 20190330 20:04:15 all (7/44): fifo4.sh mount leak: 1/25180 20190330 20:06:21 all (8/44): kevent6.sh lock order reversal: 1st 0x352823a0 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1239 2nd 0x32135d00 ufs (ufs) @ kern/vfs_subr.c:5053 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfd03d8 at lockmgr_xlock_hard+0x48 #3 0xfd0b46 at __lockmgr_args+0x376 #4 0x12cedcf at ffs_lock+0x6f #5 0x1591158 at VOP_LOCK1_APV+0x58 #6 0x10c2858 at _vn_lock+0x58 #7 0x10b3e3a at vfs_knllock+0x1a #8 0xfb3ebc at knlist_remove_kq+0x7c #9 0xfb3e33 at knlist_remove+0x13 #10 0x10b8820 at filt_vfsdetach+0x20 #11 0xfb47df at knote_fdclose+0x13f #12 0xfa8dd1 at closefp+0xe1 #13 0xfa959d at kern_close+0x12d #14 0xfa9460 at sys_close+0x10 #15 0x15616b9 at syscall+0x2d9 #16 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 lock order reversal: 1st 0x3499f028 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1239 2nd 0x30937d00 tmpfs (tmpfs) @ kern/vfs_subr.c:5053 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcffb5 at lockmgr_lock_fast_path+0x125 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0x10b3e3a at vfs_knllock+0x1a #7 0xfb3ebc at knlist_remove_kq+0x7c #8 0xfb3e33 at knlist_remove+0x13 #9 0x10b8820 at filt_vfsdetach+0x20 #10 0xfb47df at knote_fdclose+0x13f #11 0xfa8dd1 at closefp+0xe1 #12 0xfa959d at kern_close+0x12d #13 0xfa9460 at sys_close+0x10 #14 0x15616b9 at syscall+0x2d9 #15 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 mount leak: 2/25182 20190330 20:07:40 all (9/44): link.sh mount leak: 202/25384 20190330 20:12:01 all (10/44): linux.sh mount leak: 1/25385 20190330 20:12:05 all (11/44): maxvnodes.sh 20190330 20:22:35 all (12/44): md9.sh lock order reversal: 1st 0x2712359c bufwait (bufwait) @ vm/vm_pager.c:379 2nd 0x33bd01a0 tmpfs (tmpfs) @ dev/md/md.c:988 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcffb5 at lockmgr_lock_fast_path+0x125 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0xbf3ef5 at mdstart_vnode+0x375 #7 0xbf231b at md_kthread+0x1db #8 0xfbe04c at fork_exit+0x6c #9 0xffc033ca at tmpfs_rename_restarts+0xcd5f23ca mount leak: 1/25386 20190330 20:22:47 all (13/44): mkfifo5.sh mount leak: 1/25387 20190330 20:22:55 all (14/44): mkfifo6.sh mount leak: 1/25388 20190330 20:23:04 all (15/44): mkfifo7.sh mount leak: 17/25405 20190330 20:23:11 all (16/44): nfs13.sh 20190330 20:34:04 all (17/44): nullfs16.sh 20190330 20:34:16 all (18/44): nullfs9.sh 20190330 20:34:22 all (19/44): pread.sh lock order reversal: (sleepable after non-sleepable) 1st 0x26100188 pfs_node (pfs_node) @ fs/pseudofs/pseudofs_internal.h:105 2nd 0x12784814 pidhash (pidhash) @ kern/kern_proc.c:386 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0x10007d9 at _sx_slock_int+0x49 #3 0xfde64c at pfind+0x5c #4 0xf48b3a at pfs_readdir+0x13a #5 0x1590c49 at VOP_READDIR_APV+0x69 #6 0x10bfb99 at kern_getdirentries+0x1c9 #7 0x10bfdd5 at sys_getdirentries+0x25 #8 0x15616b9 at syscall+0x2d9 #9 0xffc033b7 at fdhashtbl+0xcd5eb39f 20190330 20:34:58 all (20/44): readdir.sh 20190330 20:47:06 all (21/44): rw2ro.sh 20190330 20:47:14 all (22/44): seekdir.sh 20190330 20:47:20 all (23/44): sendfile5.sh lock order reversal: 1st 0x3034db94 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x38592478 tmpfs (tmpfs) @ kern/kern_sendfile.c:699 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcfef7 at lockmgr_lock_fast_path+0x67 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0xff4912 at vn_sendfile+0x7f2 #7 0xff64c6 at sendfile+0x126 #8 0xff6390 at sys_sendfile+0x10 #9 0x15616b9 at syscall+0x2d9 #10 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 lock order reversal: 1st 0x300ea16c so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x381978bc ufs (ufs) @ kern/kern_sendfile.c:699 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfd00d5 at lockmgr_slock_hard+0x65 #3 0xfd08f6 at __lockmgr_args+0x126 #4 0x12cedcf at ffs_lock+0x6f #5 0x1591158 at VOP_LOCK1_APV+0x58 #6 0x10c2858 at _vn_lock+0x58 #7 0xff4912 at vn_sendfile+0x7f2 #8 0xff64c6 at sendfile+0x126 #9 0xff6390 at sys_sendfile+0x10 #10 0x15616b9 at syscall+0x2d9 #11 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 20190330 20:49:09 all (24/44): sendfile6.sh mount leak: 1/25406 20190330 20:49:16 all (25/44): tmpfs.sh mount leak: 1/25407 20190330 20:59:44 all (26/44): tmpfs10.sh lock order reversal: 1st 0x1fd4b00 cpslk (cpslk) @ i386/i386/copyout.c:121 2nd 0x2711759c bufwait (bufwait) @ vm/vm_pager.c:379 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfd03d8 at lockmgr_xlock_hard+0x48 #3 0xfd0b46 at __lockmgr_args+0x376 #4 0x130ccd9 at pbuf_ctor+0x79 #5 0x12e79eb at uma_zalloc_arg+0xa0b #6 0x12e0df2 at swap_pager_getpages+0x2a2 #7 0x130cf04 at vm_pager_get_pages+0x44 #8 0x12ee678 at vm_fault_hold+0x19c8 #9 0x12ecc6a at vm_fault+0x4a #10 0x15610b3 at trap_pfault+0xf3 #11 0x15606d0 at trap+0x3c0 #12 0xffc0316d at tmpfs_rename_restarts+0xcd5f216d #13 0x153c3a3 at copyin_slow0+0x13 #14 0x153c153 at cp_slow0+0x153 #15 0x153c367 at copyin+0xa7 #16 0x104eaa6 at uiomove_faultflag+0xf6 #17 0x104e9a3 at uiomove+0x13 mount leak: 1/25408 20190330 21:00:13 all (27/44): tmpfs11.sh mount leak: 1/25409 20190330 21:17:16 all (28/44): tmpfs12.sh 20190330 21:17:20 all (29/44): tmpfs13.sh mount leak: 1/25410 20190330 21:19:31 all (30/44): tmpfs14.sh 20190330 21:19:48 all (31/44): tmpfs15.sh mount leak: 1/25411 20190330 21:19:53 all (32/44): tmpfs16.sh mount leak: 1/25412 20190330 21:22:59 all (33/44): tmpfs17.sh 20190330 21:28:08 all (34/44): tmpfs18.sh mount leak: 1/25413 20190330 21:28:14 all (35/44): tmpfs19.sh mount leak: 1/25414 20190330 21:29:32 all (36/44): tmpfs2.sh mount leak: 1/25415 20190330 21:32:50 all (37/44): tmpfs3.sh 20190330 21:43:23 all (38/44): tmpfs4.sh mount leak: 1/25416 20190330 21:43:30 all (39/44): tmpfs5.sh lock order reversal: 1st 0x3cbfda28 syncer (syncer) @ kern/vfs_subr.c:2178 2nd 0x3ab75a28 tmpfs (tmpfs) @ kern/vfs_subr.c:2598 stack backtrace: #0 0x1054e11 at witness_debugger+0x61 #1 0x1054be9 at witness_checkorder+0x8a9 #2 0xfcffb5 at lockmgr_lock_fast_path+0x125 #3 0x10a1a44 at vop_stdlock+0x24 #4 0x1591158 at VOP_LOCK1_APV+0x58 #5 0x10c2858 at _vn_lock+0x58 #6 0x10b1b8b at vget+0x7b #7 0x3260a122 at tmpfs_update_mtime+0x42 #8 0x32609f04 at tmpfs_sync+0x54 #9 0x10b852f at sync_fsync+0xcf #10 0x1590219 at VOP_FSYNC_APV+0x69 #11 0x10b6577 at sched_sync+0x277 #12 0xfbe04c at fork_exit+0x6c #13 0xffc033ca at tmpfs_rename_restarts+0xcd5f23ca mount leak: 1/25417 20190330 21:43:56 all (40/44): tmpfs6.sh mount leak: 1/25418 20190330 21:44:04 all (41/44): tmpfs7.sh mount leak: 1/25419 20190330 21:44:11 all (42/44): tmpfs8.sh mount leak: 1/25420 20190330 21:45:08 all (43/44): tmpfs9.sh mount leak: 1/25421 20190330 21:57:45 all (44/44): zzbuildworld.sh Expensive timeout(9) function: 0x14724a0(0x1fa1264) 0.015504544 s 20190331 03:32:23 *** Excessive run time: zzbuildworld.sh, 274 min newblk leak: 1051/1069. mount leak: 1/25422 20190331 03:32:23 all: done panic: Assertion object->ref_count > 1 failed at /usr/src/sys/fs/tmpfs/tmpfs_vfsops.c:249 cpuid = 1 time = 1554014256 KDB: stack backtrace: db_trace_self_wrapper(2e73706f,34323a63,a39,34ce08b4,be6ee1,...) at db_trace_self_wrapper+0x2a/frame 0x34ce0888 kdb_backtrace(d,1,1,326051d1,343d9618,...) at kdb_backtrace+0x2e/frame 0x34ce08e8 vpanic(32604ec0,34ce092c,34ce092c,34ce0a14,32609b38,...) at vpanic+0x121/frame 0x34ce090c panic(32604ec0,32605528,326051d1,f9,127decf8,...) at panic+0x14/frame 0x34ce0920 tmpfs_mount(26fe7000) at tmpfs_mount+0xab8/frame 0x34ce0a14 vfs_domount(21ed0970,10001,0,34ce0bb4) at vfs_domount+0x60e/frame 0x34ce0b88 vfs_donmount(34a82a80,1,0,34a63f00) at vfs_donmount+0x79e/frame 0x34ce0be0 sys_nmount(34a82a80,34a82d08) at sys_nmount+0x5d/frame 0x34ce0c08 syscall(34ce0ce8,3b,3b,3b,20443020,...) at syscall+0x2d9/frame 0x34ce0cdc Xint0x80_syscall() at 0xffc033b7/frame 0x34ce0cdc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x204a5c7f, esp = 0xffbfd0a0, ebp = 0xffbfd5ec --- __bss_end() at 0x204a5c7f KDB: enter: panic [ thread pid 29695 tid 101425 ] Stopped at kdb_enter+0x35: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 29695 tid 101425 td 0x34a82a80 kdb_enter(16830f5,16830f5) at kdb_enter+0x35/frame 0x34ce08e0 vpanic(32604ec0,34ce092c,34ce092c,34ce0a14,32609b38,...) at vpanic+0x139/frame 0x34ce090c panic(32604ec0,32605528,326051d1,f9,127decf8,...) at panic+0x14/frame 0x34ce0920 tmpfs_mount(26fe7000) at tmpfs_mount+0xab8/frame 0x34ce0a14 vfs_domount(21ed0970,10001,0,34ce0bb4) at vfs_domount+0x60e/frame 0x34ce0b88 vfs_donmount(34a82a80,1,0,34a63f00) at vfs_donmount+0x79e/frame 0x34ce0be0 sys_nmount(34a82a80,34a82d08) at sys_nmount+0x5d/frame 0x34ce0c08 syscall(34ce0ce8,3b,3b,3b,20443020,...) at syscall+0x2d9/frame 0x34ce0cdc Xint0x80_syscall() at 0xffc033b7/frame 0x34ce0cdc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x204a5c7f, esp = 0xffbfd0a0, ebp = 0xffbfd5ec --- __bss_end() at 0x204a5c7f db:1:pho1> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x87ac80 curthread = 0x127d8700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x121e8d40 fpcurthread = none idlethread = 0x127d8700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x121e8d30 common_tssp = 0xffc01260 tlb gen = 19431005 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x109dac80 curthread = 0x34a82a80: pid 29695 tid 101425 "mount" curpcb = 0x34ce0d40 fpcurthread = 0x34a82a80: pid 29695 "mount" idlethread = 0x127d8380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x34ce0d30 common_tssp = 0xffc012c8 tlb gen = 19431001 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x109dfc80 curthread = 0x2fa39000: pid 560 tid 100112 "mountd" curpcb = 0x2f8d3d40 fpcurthread = 0x2fa39000: pid 560 "mountd" idlethread = 0x127d8000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x2f8d3d30 common_tssp = 0xffc01330 tlb gen = 19431005 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x109e4c80 curthread = 0x127d7a80: pid 11 tid 100006 "idle: cpu3" curpcb = 0x121f7d40 fpcurthread = none idlethread = 0x127d7a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x121f7d30 common_tssp = 0xffc01398 tlb gen = 19431005 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 29695 (mount) thread 0x34a82a80 (101425) shared rw vm object (vm object) r = 0 (0x343d9618) locked @ /usr/src/sys/fs/tmpfs/tmpfs_vfsops.c:244 exclusive sx vm map (user) (vm map (user)) r = 0 (0x127ded38) locked @ /usr/src/sys/fs/tmpfs/tmpfs_vfsops.c:224 Process 560 (mountd) thread 0x2fa39000 (100112) exclusive lockmgr ufs (ufs) r = 0 (0x3a786750) locked @ kern/vfs_vnops.c:449 db:1:pho1> show lockedvnods Locked vnodes vnode 0x3a78671c: tag ufs, type VREG usecount 0, writecount 0, refcount 3 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x419544e0 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x2fa39000 (pid 560, mountd, tid 100112) #0 0xfd0331 at lockmgr_slock_hard+0x2c1 #1 0xfd08f6 at __lockmgr_args+0x126 #2 0x12cedcf at ffs_lock+0x6f #3 0x1591158 at VOP_LOCK1_APV+0x58 #4 0x10c2858 at _vn_lock+0x58 #5 0x10c2976 at vn_close1+0x56 #6 0x10c1845 at vn_closefile+0x45 #7 0xfa8948 at _fdrop+0x18 #8 0xfab530 at closef+0x1e0 #9 0xfa8d6d at closefp+0x7d #10 0xfa959d at kern_close+0x12d #11 0xfa9460 at sys_close+0x10 #12 0x15616b9 at syscall+0x2d9 #13 0xffc033b7 at tmpfs_rename_restarts+0xcd5f23b7 nlink=1, effnlink=1, size=276, extsize 0 generation=dd7824fa, uid=0, gid=0, flags=0x0 ino 23559, on dev ada0s1a db:1:pho1> show allchains chain 1: thread 101527 (pid 29689, sleep) sleeping on 0x1f0fd7d "nanslp" chain 2: thread 101686 (pid 28936, sh) sleeping on 0x3aad0000 "wait" chain 3: thread 100183 (pid 1229, sh) sleeping on 0x2fa49b98 "piperd" chain 4: thread 100210 (pid 1227, awk) sleeping on 0x30269350 "piperd" chain 5: thread 100190 (pid 1226, sh) sleeping on 0x260fd000 "wait" chain 6: thread 100177 (pid 1205, sh) blocked on lockmgr 00SHARED (count 1474196) chain 7: thread 100201 (pid 809, bash) sleeping on 0x30b40a08 "wait" chain 8: thread 100198 (pid 807, csh) sleeping on 0x30bfc714 "pause" chain 9: thread 100175 (pid 806, su) sleeping on 0x26103a08 "wait" chain 10: thread 100124 (pid 793, bash) blocked on lockmgr X0/SHARED (count 1474196) chain 11: thread 100199 (pid 792, sshd) sleeping on 0x26b33964 "select" chain 12: thread 100181 (pid 781, sshd) sleeping on 0x26b34ca4 "select" chain 13: thread 100197 (pid 779, getty) sleeping on 0x265ec474 "ttyin" chain 14: thread 100196 (pid 778, getty) sleeping on 0x26ccd874 "ttyin" chain 15: thread 100195 (pid 777, getty) sleeping on 0x26ccdc74 "ttyin" chain 16: thread 100194 (pid 776, getty) sleeping on 0x26cce074 "ttyin" chain 17: thread 100193 (pid 775, getty) sleeping on 0x26cce474 "ttyin" chain 18: thread 100192 (pid 774, getty) sleeping on 0x26cce874 "ttyin" chain 19: thread 100191 (pid 773, getty) sleeping on 0x26ccec74 "ttyin" chain 20: thread 100188 (pid 772, getty) sleeping on 0x26ccf074 "ttyin" chain 21: thread 100105 (pid 771, getty) sleeping on 0x26ccf474 "ttyin" chain 22: thread 100184 (pid 754, inetd) sleeping on 0x12354824 "select" chain 23: thread 100114 (pid 720, nfscbd: master) sleeping on 0x26cd1dd0 "rpcsvc" chain 24: thread 100185 (pid 720, nfscbd: service) sleeping on 0x26e947d0 "rpcsvc" chain 25: thread 100186 (pid 720, nfscbd: service) sleeping on 0x26e94850 "rpcsvc" chain 26: thread 100187 (pid 720, nfscbd: service) sleeping on 0x26e948d0 "rpcsvc" chain 27: thread 100182 (pid 719, nfscbd) sleeping on 0x300798e8 "accept" chain 28: thread 100119 (pid 686, cron) sleeping on 0x1f0fd7d "nanslp" chain 29: thread 100174 (pid 682, sendmail) sleeping on 0x30b2d3bc "pause" chain 30: thread 100111 (pid 679, sendmail) sleeping on 0x26b347e4 "select" chain 31: thread 100120 (pid 676, sshd) sleeping on 0x26b346e4 "select" chain 32: thread 100171 (pid 636, smartd) sleeping on 0x1f0fd7e "nanslp" chain 33: thread 100170 (pid 612, powerd) sleeping on 0x26b353e4 "select" chain 34: thread 100169 (pid 609, ntpd) sleeping on 0x26b34824 "select" chain 35: thread 100167 (pid 573, NLM: master) sleeping on 0x2fb6f650 "rpcsvc" chain 36: thread 100166 (pid 570, rpc.statd) sleeping on 0x26b353a4 "select" chain 37: thread 100106 (pid 569, nfsd: master) sleeping on 0x26e93e50 "rpcsvc" chain 38: thread 100135 (pid 569, nfsd: service) sleeping on 0x26e93ed0 "rpcsvc" chain 39: thread 100136 (pid 569, nfsd: service) sleeping on 0x123bd150 "rpcsvc" chain 40: thread 100137 (pid 569, nfsd: service) sleeping on 0x2fb6fd50 "rpcsvc" chain 41: thread 100138 (pid 569, nfsd: service) sleeping on 0x123bb5d0 "rpcsvc" chain 42: thread 100139 (pid 569, nfsd: service) sleeping on 0x2fb6fcd0 "rpcsvc" chain 43: thread 100140 (pid 569, nfsd: service) sleeping on 0x123bb6d0 "rpcsvc" chain 44: thread 100141 (pid 569, nfsd: service) sleeping on 0x123bb750 "rpcsvc" chain 45: thread 100142 (pid 569, nfsd: service) sleeping on 0x123bd1d0 "rpcsvc" chain 46: thread 100143 (pid 569, nfsd: service) sleeping on 0x2fb6fc50 "rpcsvc" chain 47: thread 100144 (pid 569, nfsd: service) sleeping on 0x123bb7d0 "rpcsvc" chain 48: thread 100145 (pid 569, nfsd: service) sleeping on 0x123bb850 "rpcsvc" chain 49: thread 100146 (pid 569, nfsd: service) sleeping on 0x123bb8d0 "rpcsvc" chain 50: thread 100147 (pid 569, nfsd: service) sleeping on 0x2fb6fbd0 "rpcsvc" chain 51: thread 100148 (pid 569, nfsd: service) sleeping on 0x2fb6fb50 "rpcsvc" chain 52: thread 100149 (pid 569, nfsd: service) sleeping on 0x2fb6fad0 "rpcsvc" chain 53: thread 100150 (pid 569, nfsd: service) sleeping on 0x123bb950 "rpcsvc" chain 54: thread 100151 (pid 569, nfsd: service) sleeping on 0x123bd250 "rpcsvc" chain 55: thread 100152 (pid 569, nfsd: service) sleeping on 0x2fb6fa50 "rpcsvc" chain 56: thread 100153 (pid 569, nfsd: service) sleeping on 0x123bb9d0 "rpcsvc" chain 57: thread 100154 (pid 569, nfsd: service) sleeping on 0x2fb6f9d0 "rpcsvc" chain 58: thread 100155 (pid 569, nfsd: service) sleeping on 0x2fb6f950 "rpcsvc" chain 59: thread 100156 (pid 569, nfsd: service) sleeping on 0x123bba50 "rpcsvc" chain 60: thread 100157 (pid 569, nfsd: service) sleeping on 0x2fb6f8d0 "rpcsvc" chain 61: thread 100158 (pid 569, nfsd: service) sleeping on 0x2fb6f850 "rpcsvc" chain 62: thread 100159 (pid 569, nfsd: service) sleeping on 0x26e94550 "rpcsvc" chain 63: thread 100160 (pid 569, nfsd: service) sleeping on 0x2fb6f7d0 "rpcsvc" chain 64: thread 100161 (pid 569, nfsd: service) sleeping on 0x2fb6f750 "rpcsvc" chain 65: thread 100162 (pid 569, nfsd: service) sleeping on 0x26e945d0 "rpcsvc" chain 66: thread 100163 (pid 569, nfsd: service) sleeping on 0x2fb6f6d0 "rpcsvc" chain 67: thread 100164 (pid 569, nfsd: service) sleeping on 0x26e94650 "rpcsvc" chain 68: thread 100165 (pid 569, nfsd: service) sleeping on 0x26e946d0 "rpcsvc" chain 69: thread 100108 (pid 566, nfsd) sleeping on 0x26b35524 "select" chain 70: thread 100134 (pid 553, nfsuserd) sleeping on 0x12354aa4 "select" chain 71: thread 100133 (pid 552, nfsuserd) sleeping on 0x123549a4 "select" chain 72: thread 100130 (pid 551, nfsuserd) sleeping on 0x26b347a4 "select" chain 73: thread 100107 (pid 550, nfsuserd) sleeping on 0x30290928 "sbwait" chain 74: thread 100129 (pid 549, nfsuserd) sleeping on 0x30051a6c "pause" chain 75: thread 100125 (pid 506, watchdogd) sleeping on 0x1f0fd7f "nanslp" chain 76: thread 100127 (pid 421, rpcbind) sleeping on 0x26b35364 "select" chain 77: thread 100121 (pid 412, syslogd) sleeping on 0x26b357a4 "select" chain 78: thread 100122 (pid 340, devd) sleeping on 0x26b34724 "select" chain 79: thread 100104 (pid 25, vnlru) sleeping on 0x127d3000 "vlruwt" chain 80: thread 100103 (pid 24, syncer) sleeping on 0x1f8a4e4 "syncer" chain 81: thread 100100 (pid 23, bufdaemon) sleeping on 0x1f89e84 "qsleep" chain 82: thread 100102 (pid 23, bufspacedaemon-0) sleeping on 0x1c3d280 "-" chain 83: thread 100115 (pid 23, /home worker) sleeping on 0x2712aa84 "sdflush" chain 84: thread 100116 (pid 23, /tmp worker) sleeping on 0x21f0c084 "sdflush" chain 85: thread 100117 (pid 23, /usr worker) sleeping on 0x21f0be84 "sdflush" chain 86: thread 100118 (pid 23, /var worker) sleeping on 0x21f0bc84 "sdflush" chain 87: thread 100098 (pid 22, vmdaemon) sleeping on 0x1f9fa38 "psleep" chain 88: thread 100097 (pid 21, dom0) sleeping on 0x1fd3ff8 "psleep" chain 89: thread 100099 (pid 21, laundry: dom0) sleeping on 0x1fd4004 "launds" chain 90: thread 100101 (pid 21, uma) sleeping on 0x12e9bc0 "umarcl" chain 91: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1d75240 "-" chain 92: thread 100090 (pid 19, sctp_iterator) sleeping on 0x1fd30bc "waiting_for_work" chain 93: thread 100089 (pid 18, soaiod4) sleeping on 0x1f89aa8 "-" chain 94: thread 100088 (pid 17, soaiod3) sleeping on 0x1f89aa8 "-" chain 95: thread 100087 (pid 16, soaiod2) sleeping on 0x1f89aa8 "-" chain 96: thread 100086 (pid 9, soaiod1) sleeping on 0x1f89aa8 "-" chain 97: thread 100083 (pid 8, fdc0) sleeping on 0x21f4223c "-" chain 98: thread 100048 (pid 15, usbus0) sleeping on 0x21f7dacc "-" chain 99: thread 100049 (pid 15, usbus0) sleeping on 0x21f7dafc "-" chain 100: thread 100050 (pid 15, usbus0) sleeping on 0x21f7db2c "-" chain 101: thread 100051 (pid 15, usbus0) sleeping on 0x21f7db5c "-" chain 102: thread 100052 (pid 15, usbus0) sleeping on 0x21f7db8c "-" chain 103: thread 100054 (pid 15, usbus1) sleeping on 0x26108acc "-" chain 104: thread 100055 (pid 15, usbus1) sleeping on 0x26108afc "-" chain 105: thread 100056 (pid 15, usbus1) sleeping on 0x26108b2c "-" chain 106: thread 100057 (pid 15, usbus1) sleeping on 0x26108b5c "-" chain 107: thread 100058 (pid 15, usbus1) sleeping on 0x26108b8c "-" chain 108: thread 100060 (pid 15, usbus2) sleeping on 0x26156acc "-" chain 109: thread 100061 (pid 15, usbus2) sleeping on 0x26156afc "-" chain 110: thread 100062 (pid 15, usbus2) sleeping on 0x26156b2c "-" chain 111: thread 100063 (pid 15, usbus2) sleeping on 0x26156b5c "-" chain 112: thread 100064 (pid 15, usbus2) sleeping on 0x26156b8c "-" chain 113: thread 100065 (pid 15, usbus3) sleeping on 0x261a4acc "-" chain 114: thread 100066 (pid 15, usbus3) sleeping on 0x261a4afc "-" chain 115: thread 100067 (pid 15, usbus3) sleeping on 0x261a4b2c "-" chain 116: thread 100068 (pid 15, usbus3) sleeping on 0x261a4b5c "-" chain 117: thread 100069 (pid 15, usbus3) sleeping on 0x261a4b8c "-" chain 118: thread 100070 (pid 15, usbus4) sleeping on 0x261eaacc "-" chain 119: thread 100071 (pid 15, usbus4) sleeping on 0x261eaafc "-" chain 120: thread 100072 (pid 15, usbus4) sleeping on 0x261eab2c "-" chain 121: thread 100073 (pid 15, usbus4) sleeping on 0x261eab5c "-" chain 122: thread 100074 (pid 15, usbus4) sleeping on 0x261eab8c "-" chain 123: thread 100076 (pid 15, usbus5) sleeping on 0x21f82ca4 "-" chain 124: thread 100077 (pid 15, usbus5) sleeping on 0x21f82cd4 "-" chain 125: thread 100078 (pid 15, usbus5) sleeping on 0x21f82d04 "-" chain 126: thread 100079 (pid 15, usbus5) sleeping on 0x21f82d34 "-" chain 127: thread 100080 (pid 15, usbus5) sleeping on 0x21f82d64 "-" chain 128: thread 100043 (pid 7, doneq0) sleeping on 0x1cae640 "-" chain 129: thread 100096 (pid 7, scanner) sleeping on 0x1cae56c "-" chain 130: thread 100042 (pid 14, sequencer 00) sleeping on 0x21e3624c "seqstate" chain 131: thread 100041 (pid 6, crypto returns 3) sleeping on 0x21e280c4 "crypto_ret_wait" chain 132: thread 100040 (pid 5, crypto returns 2) sleeping on 0x21e2808c "crypto_ret_wait" chain 133: thread 100039 (pid 4, crypto returns 1) sleeping on 0x21e28054 "crypto_ret_wait" chain 134: thread 100038 (pid 3, crypto returns 0) sleeping on 0x21e2801c "crypto_ret_wait" chain 135: thread 100037 (pid 2, crypto) sleeping on 0x1f9e800 "crypto_wait" chain 136: thread 100029 (pid 13, g_event) sleeping on 0x1fc307c "-" chain 137: thread 100030 (pid 13, g_up) sleeping on 0x1fc3084 "-" chain 138: thread 100031 (pid 13, g_down) sleeping on 0x1fc3078 "-" chain 139: thread 100002 (pid 1, init) sleeping on 0x127d4a08 "wait" chain 140: thread 100001 (pid 10, audit) sleeping on 0x1fd38a0 "audit_worker_cv" chain 141: thread 100000 (pid 0, swapper) sleeping on 0x1fc309c "swapin" chain 142: thread 100007 (pid 0, aiod_kick taskq) sleeping on 0x122d8000 "-" chain 143: thread 100010 (pid 0, thread taskq) sleeping on 0x122d7e00 "-" chain 144: thread 100012 (pid 0, config_0) sleeping on 0x1223c800 "-" chain 145: thread 100013 (pid 0, kqueue_ctx taskq) sleeping on 0x122d7d00 "-" chain 146: thread 100014 (pid 0, if_config_tqg_0) sleeping on 0x1223c780 "-" chain 147: thread 100015 (pid 0, softirq_0) sleeping on 0x1223c100 "-" chain 148: thread 100016 (pid 0, softirq_1) sleeping on 0x1223c080 "-" chain 149: thread 100017 (pid 0, softirq_2) sleeping on 0x1223c000 "-" chain 150: thread 100018 (pid 0, softirq_3) sleeping on 0x1223bf00 "-" chain 151: thread 100019 (pid 0, if_io_tqg_0) sleeping on 0x1223be80 "-" chain 152: thread 100020 (pid 0, if_io_tqg_1) sleeping on 0x1223be00 "-" chain 153: thread 100021 (pid 0, if_io_tqg_2) sleeping on 0x1223bd80 "-" chain 154: thread 100022 (pid 0, if_io_tqg_3) sleeping on 0x1223bd00 "-" chain 155: thread 100032 (pid 0, firmware taskq) sleeping on 0x123d8080 "-" chain 156: thread 100033 (pid 0, crypto_0) sleeping on 0x123d8000 "-" chain 157: thread 100034 (pid 0, crypto_1) sleeping on 0x123d8000 "-" chain 158: thread 100035 (pid 0, crypto_2) sleeping on 0x123d8000 "-" chain 159: thread 100036 (pid 0, crypto_3) sleeping on 0x123d8000 "-" chain 160: thread 100085 (pid 0, mca taskq) sleeping on 0x21f57c00 "-" chain 161: thread 100091 (pid 0, acpi_task_0) sleeping on 0x21f57a00 "-" chain 162: thread 100092 (pid 0, acpi_task_1) sleeping on 0x21f57a00 "-" chain 163: thread 100093 (pid 0, acpi_task_2) sleeping on 0x21f57a00 "-" chain 164: thread 100095 (pid 0, CAM taskq) sleeping on 0x123d7f00 "-" db:1:pho1> show mount 0x27181000 /dev/ada0s1a on / (ufs) 0x26fe9000 devfs on /dev (devfs) 0x2fb83000 /dev/ada0s1f on /home (ufs) 0x2fb82000 /dev/label/tmp on /tmp (ufs) 0x2fb81000 /dev/ada0s1d on /usr (ufs) 0x2fbcf000 /dev/ada0s1e on /var (ufs) 0x2fbce000 procfs on /proc (procfs) 0x26fe7000 tmpfs on /mnt (tmpfs) More info: show mount db:1:pho1> show bufqueues bqempty: 0 Buf domain 0 freebufs 459 lofreebufs 311 hifreebufs 466 bufspace 82958336 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 42 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5243 (81854464) dirtyq count 42 (665600) wakeup 0 lim 28 CPU 1, 20, 2, 12, Locked buffers: 0 space 0 Total buffers: 5779 space 82958336 db:1:pho1> show page vm_cnt.v_free_count: 75803 vm_cnt.v_inactive_count: 1847495 vm_cnt.v_active_count: 972 vm_cnt.v_laundry_count: 141 Loading /boot/loader.conf.local -- Test scenario: misc/rw2ro.sh + f1=/mnt/f1 + f2=/mnt/f2 + s=0 + mount -o 'size=100m' -t tmpfs null /mnt + touch /mnt/f1 + mount -u -o ro /mnt