KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #1 r347532M: Wed May 15 01:12:54 CEST 2019 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3516727296 (3353 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) __stack_chk_init: WARNING: Initializing stack protection with non-random cookies! __stack_chk_init: WARNING: This severely limits the benefit of -fstack-protector! ioapic0 irqs 0-23 on motherboard Launching APs: 1 3 2 Timecounter "TSC" frequency 1800013330 Hz quality 800 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0x14c97d0, 0) error 19 kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 vgapci0: Boot video device hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahci0: quirks=0x7000 ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1 on ohci1 usbus1: 12Mbps Full Speed USB v1.0 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2 on ohci2 usbus2: 12Mbps Full Speed USB v1.0 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3 on ohci3 usbus3: 12Mbps Full Speed USB v1.0 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4 on ohci4 usbus4: 12Mbps Full Speed USB v1.0 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5 on ehci0 usbus5: 480Mbps High Speed USB v2.0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pci3: at device 6.0 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. hwpstate0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 3 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 20,22,21,23 and 24,26 on hdaa1 pcm2: at nid 27 and 25 on hdaa1 pcm3: at nid 30 on hdaa1 ugen0.1: at usbus0 ugen5.1: at usbus5 ugen3.1: at usbus3 uhub0: on usbus0 ugen4.1: at usbus4 uhub1: on usbus5 uhub2: on usbus3 ugen2.1: at usbus2 uhub3: on usbus4 Expensive timeout(9) function: 0xa3aa90(0x17b45000) 0.059329328 s uhub4: on usbus2 ugen1.1: at usbus1 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 uhub5: on usbus1 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) cd0 at ata0 bus 0 scbus4 target 1 lun 0 cd0: <_NEC DVD_RW ND-4550A 1.06> Removable CD-ROM SCSI device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Deprecated code (to be removed in FreeBSD 14): FreeBSD 12.x ABI compat Deprecated code (to be removed in FreeBSD 14): FreeBSD 12.x ABI compat warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 610435 free (2651 frags, 75973 blocks, 0.3% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 777438 free (10974 frags, 95808 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 28350600 free (36880 frags, 3539215 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1470483 free (294203 frags, 147035 blocks, 2.9% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2941721 free (168313 frags, 346676 blocks, 0.7% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /urandom: unblocking device. sr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: uhub1: 10 ports with 10 removable, self powered . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0x96614a0 bufwait (bufwait) @ kern/vfs_bio.c:3903 2nd 0x9470400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0x10015dd at _sx_xlock+0x4d #3 0x12c88e5 at ufsdirhash_add+0x45 #4 0x12cb15e at ufs_direnter+0x3fe #5 0x12d2c0c at ufs_mkdir+0x78c #6 0x1587f9d at VOP_MKDIR_APV+0x6d #7 0x10c1913 at kern_mkdirat+0x173 #8 0x10c1799 at sys_mkdir+0x19 #9 0x1558d79 at syscall+0x2d9 #10 0xffc033b7 at _end+0xfdc168c7 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. May 15 01:16:45 x4 smartd[622]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, May 15 01:16:45 x4 smartd[622]: see the following Seagate web pages: May 15 01:16:45 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en May 15 01:16:45 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Wed May 15 01:16 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: May 15 01:17:42 x4 su[904]: pho to root on /dev/pts/0 20190515 01:33:42 all (1/1): su.sh lock order reversal: 1st 0x1fcdb00 cpslk (cpslk) @ i386/i386/copyout.c:121 2nd 0x18957c9c bufwait (bufwait) @ vm/vm_pager.c:379 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1db8 at lockmgr_xlock_hard+0x48 #3 0xfd2526 at __lockmgr_args+0x376 #4 0x1301839 at pbuf_ctor+0x79 #5 0x12dc43b at uma_zalloc_arg+0xa0b #6 0x12d5942 at swap_pager_getpages+0x2a2 #7 0x1301a64 at vm_pager_get_pages+0x44 #8 0x12e30c8 at vm_fault_hold+0x19c8 #9 0x12e16ba at vm_fault+0x4a #10 0x1558773 at trap_pfault+0xf3 #11 0x1557d90 at trap+0x3c0 #12 0xffc0316d at _end+0xfdc1667d #13 0x1533923 at copyin_slow0+0x13 #14 0x15336d3 at cp_slow0+0x153 #15 0x15338e7 at copyin+0xa7 #16 0x10510e6 at uiomove_faultflag+0xf6 #17 0x1050fe3 at uiomove+0x13 20190515 01:36:53 all: done 20190515 01:40:32 all (1/620): mac.sh 20190515 01:40:35 all (2/620): parallelmount2.sh 20190515 01:40:37 all (3/620): 256m.sh 20190515 01:40:40 all (4/620): ldt2.sh 20190515 01:40:42 all (5/620): linux.sh 20190515 01:40:45 all (6/620): pkru2.sh 20190515 01:41:22 all (1/620): mac.sh 20190515 01:41:25 all (2/620): parallelmount2.sh 20190515 01:41:28 all (3/620): 256m.sh 20190515 01:41:31 all (4/620): ldt2.sh 20190515 01:41:33 all (5/620): linux.sh 20190515 01:41:36 all (6/620): pkru2.sh 20190515 01:41:39 all (7/620): schedfuzz.sh 20190515 01:41:42 all (8/620): tmpfs12.sh 20190515 01:41:45 all (9/620): trim6.sh 20190515 01:41:47 all (10/620): maxmemdom.sh 20190515 01:41:51 all (11/620): md4.sh 20190515 01:41:54 all (12/620): uma_zalloc_arg.sh 20190515 01:41:58 all (13/620): 1st.sh 20190515 01:42:02 all (14/620): audit.sh 20190515 01:42:05 all (15/620): buildworld2.sh 20190515 01:42:09 all (16/620): dumpfs.sh lock order reversal: 1st 0x18946c9c bufwait (bufwait) @ vm/vm_pager.c:379 2nd 0x296171a0 ufs (ufs) @ dev/md/md.c:996 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1db8 at lockmgr_xlock_hard+0x48 #3 0xfd2526 at __lockmgr_args+0x376 #4 0x12c339f at ffs_lock+0x6f #5 0x1588948 at VOP_LOCK1_APV+0x58 #6 0x10c4c98 at _vn_lock+0x58 #7 0xbf4b87 at mdstart_vnode+0x4b7 #8 0xbf2e6b at md_kthread+0x1db #9 0xfbf9ac at fork_exit+0x6c #10 0xffc033ca at _end+0xfdc168da lock order reversal: 1st 0x21c61a28 ufs (ufs) @ kern/vfs_mount.c:1353 2nd 0x295d0034 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1394 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1995 at lockmgr_lock_fast_path+0x125 #3 0x10a3fb4 at vop_stdlock+0x24 #4 0x1588948 at VOP_LOCK1_APV+0x58 #5 0x10c4c98 at _vn_lock+0x58 #6 0x12bec93 at ffs_flushfiles+0xb3 #7 0x12a4651 at softdep_flushfiles+0x61 #8 0x12c11b2 at ffs_unmount+0x72 #9 0x10ad195 at dounmount+0x415 #10 0x10acc8b at sys_unmount+0x26b #11 0x1558d79 at syscall+0x2d9 #12 0xffc033b7 at _end+0xfdc168c7 newblk leak: 637/647. 20190515 01:42:14 all (17/620): extattrctl.sh 20190515 01:42:18 all (18/620): msdos2.sh mount leak: 1/98 20190515 01:42:21 all (19/620): msdos3.sh mount leak: 1/99 20190515 01:42:25 all (20/620): nullfs14.sh mount leak: 1/100 20190515 01:42:29 all (21/620): nullfs6.sh acquiring duplicate lock of same type: "vnode interlock" 1st vnode interlock @ /usr/src/sys/fs/nullfs/null_vnops.c:342 2nd vnode interlock @ kern/vfs_default.c:1116 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfdb216 at __mtx_lock_flags+0x76 #3 0x10a5144 at vop_stdadd_writecount+0x24 #4 0x158b4af at VOP_ADD_WRITECOUNT_APV+0x6f #5 0x29803ca0 at null_add_writecount+0x70 #6 0x158b4af at VOP_ADD_WRITECOUNT_APV+0x6f #7 0x10c4bb0 at vn_open_vnode+0x2d0 #8 0x10c4712 at vn_open_cred+0x2e2 #9 0x10c4420 at vn_open+0x20 #10 0x10bcff2 at kern_openat+0x242 #11 0x10bd22f at sys_openat+0x2f #12 0x1558d79 at syscall+0x2d9 #13 0xffc033b7 at null_bug_bypass+0xd63fd39b mount leak: 1/101 20190515 01:42:33 all (22/620): nullfs9.sh 20190515 01:42:37 all (23/620): tmpfs15.sh mount leak: 1/102 20190515 01:42:41 all (24/620): mkfifo7.sh freework leak: 1/9. mount leak: 17/119 20190515 01:42:51 all (25/620): mprotect.sh 20190515 01:42:55 all (26/620): pagefault.sh freework leak: 12/21. 20190515 01:42:58 all (27/620): tmpfs4.sh 20190515 01:43:04 all (28/620): mmap8.sh 20190515 01:43:09 all (29/620): mountro4.sh lock order reversal: 1st 0x2502b034 msdosfs (msdosfs) @ kern/vfs_subr.c:3112 2nd 0x29aa9750 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:854 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1995 at lockmgr_lock_fast_path+0x125 #3 0x10a3fb4 at vop_stdlock+0x24 #4 0x1588948 at VOP_LOCK1_APV+0x58 #5 0x10c4c98 at _vn_lock+0x58 #6 0xedecf3 at msdosfs_fsync+0x43 #7 0x1587a09 at VOP_FSYNC_APV+0x69 #8 0x10b4bcb at vflush+0x20b #9 0xedc334 at msdosfs_mount+0xa4 #10 0x10abefe at vfs_domount+0x60e #11 0x10ab3de at vfs_donmount+0x79e #12 0x10aac0d at sys_nmount+0x5d #13 0x1558d79 at syscall+0x2d9 #14 0xffc033b7 at tmpfs_rename_restarts+0xd63f23b7 20190515 01:43:16 all (30/620): tmpfs6.sh 20190515 01:43:22 all (31/620): mmap.sh 20190515 01:43:26 all (32/620): mmap26.sh 20190515 01:43:31 all (33/620): extattr_set_fd.sh 20190515 01:43:36 all (34/620): mlockall5.sh 20190515 01:43:40 all (35/620): msdos8.sh 20190515 01:43:44 all (36/620): nanosleep.sh 20190515 01:43:48 all (37/620): ptrace8.sh 20190515 01:43:53 all (38/620): sigstop.sh 20190515 01:43:58 all (39/620): unload.sh 20190515 01:44:02 all (40/620): mmap24.sh 20190515 01:44:07 all (41/620): sendfile6.sh lock order reversal: 1st 0x218d698c so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x29aaad00 ufs (ufs) @ kern/kern_sendfile.c:699 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1ab5 at lockmgr_slock_hard+0x65 #3 0xfd22d6 at __lockmgr_args+0x126 #4 0x12c339f at ffs_lock+0x6f #5 0x1588948 at VOP_LOCK1_APV+0x58 #6 0x10c4c98 at _vn_lock+0x58 #7 0xff6432 at vn_sendfile+0x7f2 #8 0xff7fe6 at sendfile+0x126 #9 0xff7eb0 at sys_sendfile+0x10 #10 0x1558d79 at syscall+0x2d9 #11 0xffc033b7 at _end+0xfdc168c7 20190515 01:44:11 all (42/620): tmpfs7.sh lock order reversal: 1st 0x218d6784 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x29aaa750 tmpfs (tmpfs) @ kern/kern_sendfile.c:699 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd18d7 at lockmgr_lock_fast_path+0x67 #3 0x10a3fb4 at vop_stdlock+0x24 #4 0x1588948 at VOP_LOCK1_APV+0x58 #5 0x10c4c98 at _vn_lock+0x58 #6 0xff6432 at vn_sendfile+0x7f2 #7 0xff7fe6 at sendfile+0x126 #8 0xff7eb0 at sys_sendfile+0x10 #9 0x1558d79 at syscall+0x2d9 #10 0xffc033b7 at tmpfs_rename_restarts+0xd63f23b7 20190515 01:44:16 all (43/620): truncate6.sh 20190515 01:44:20 all (44/620): vfork.sh 20190515 01:44:25 all (45/620): md9.sh lock order reversal: 1st 0x18946c9c bufwait (bufwait) @ vm/vm_pager.c:379 2nd 0x29ac7478 tmpfs (tmpfs) @ dev/md/md.c:991 stack backtrace: #0 0x1057451 at witness_debugger+0x61 #1 0x1057229 at witness_checkorder+0x8a9 #2 0xfd1995 at lockmgr_lock_fast_path+0x125 #3 0x10a3fb4 at vop_stdlock+0x24 #4 0x1588948 at VOP_LOCK1_APV+0x58 #5 0x10c4c98 at _vn_lock+0x58 #6 0xbf4a45 at mdstart_vnode+0x375 #7 0xbf2e6b at md_kthread+0x1db #8 0xfbf9ac at fork_exit+0x6c #9 0xffc033ca at tmpfs_rename_restarts+0xd63f23ca 20190515 01:44:34 all (46/620): mmap4.sh vnode_pager_putpages: I/O error 28 0x29aac5b0: tag ufs, type VREG usecount 0, writecount 0, refcount 2679 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x21e653a8 ref 0 pages 25608 cleanbuf 2676 dirtybuf 1 lock type ufs: EXCL by thread 0x227e6380 (pid 13312, mmap4, tid 100247) #0 0xfd2087 at lockmgr_xlock_hard+0x317 #1 0xfd2526 at __lockmgr_args+0x376 #2 0x12c339f at ffs_lock+0x6f #3 0x1588948 at VOP_LOCK1_APV+0x58 #4 0x10c4c98 at _vn_lock+0x58 #5 0x10b4679 at vputx+0x179 #6 0x12f2469 at vm_object_deallocate+0x489 #7 0x12e7a6e at vm_map_process_deferred+0x6e #8 0x12ecd96 at vm_map_remove+0xa6 #9 0x12e767c at vmspace_exit+0xcc #10 0xfbaa80 at exit1+0x510 #11 0xfba562 at sys_sys_exit+0x12 #12 0x1558d79 at syscall+0x2d9 #13 0xffc033b7 at _end+0xfdc168c7 nlink=1, effnlink=1, size=104857600, extsize 0 generation=2ef8f38e, uid=0, gid=0, flags=0x0 ino 4, on dev md10a vnode_pager_putpages: residual I/O 32768 at 9760 0x29aac5b0: tag ufs, type VREG usecount 0, writecount 0, refcount 2679 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x21e653a8 ref 0 pages 25608 cleanbuf 2676 dirtybuf 1 lock type ufs: EXCL by thread 0x227e6380 (pid 13312, mmap4, tid 100247) #0 0xfd2087 at lockmgr_xlock_hard+0x317 #1 0xfd2526 at __lockmgr_args+0x376 #2 0x12c339f at ffs_lock+0x6f #3 0x1588948 at VOP_LOCK1_APV+0x58 #4 0x10c4c98 at _vn_lock+0x58 #5 0x10b4679 at vputx+0x179 #6 0x12f2469 at vm_object_deallocate+0x489 #7 0x12e7a6e at vm_map_process_deferred+0x6e #8 0x12ecd96 at vm_map_remove+0xa6 #9 0x12e767c at vmspace_exit+0xcc #10 0xfbaa80 at exit1+0x510 #11 0xfba562 at sys_sys_exit+0x12 #12 0x1558d79 at syscall+0x2d9 #13 0xffc033b7 at _end+0xfdc168c7 nlink=1, effnlink=1, size=104857600, extsize 0 generation=2ef8f38e, uid=0, gid=0, flags=0x0 ino 4, on dev md10aMay 15 01:44:37 x4 kernel: pid 13312 (mmap4), uid 0 inumber 4 on /mnt: filesystem full vnode_pager_putpages: zero-length write at 40075264 resid 131072 0x29aac5b0: tag ufs, type VREG usecount 1, writecount 0, refcount 2609 flags (VI_ACTIVE) v_object 0x21e653a8 ref 0 pages 25608 cleanbuf 2607 dirtybuf 0 lock type ufs: EXCL by thread 0x25714700 (pid 14467, umount, tid 100302) #0 0xfd2087 at lockmgr_xlock_hard+0x317 #1 0xfd2526 at __lockmgr_args+0x376 #2 0x12c339f at ffs_lock+0x6f #3 0x1588948 at VOP_LOCK1_APV+0x58 #4 0x10c4c98 at _vn_lock+0x58 #5 0x10b3e3b at vget+0x7b #6 0x10b5d52 at vfs_msync+0x92 #7 0x10ad11c at dounmount+0x39c #8 0x10acc8b at sys_unmount+0x26b May 15 01:44:39 #9 0x1558d79 at syscall+0x2d9 x4 kernel: pid 1#10 0xffc033b7 at _end+0xfdc168c7 nlink=1, effnlink=1, size=104857600, extsize 0 generation=2ef8f38e, uid=0, gid=0, flags=0x0 ino 4, on dev md10a 4467 (umount), uid 0 inumber 4 on /mnt: filesystem full 20190515 01:44:41 all (47/620): nullfs16.sh 20190515 01:44:51 all (48/620): kevent10.sh 20190515 01:44:55 all (49/620): extattr.sh 20190515 01:45:04 all (50/620): mkfifo5.sh mount leak: 1/120 20190515 01:45:08 all (51/620): msdos9.sh mount leak: 1/121 20190515 01:45:17 all (52/620): trim2.sh mount leak: 1/122 20190515 01:45:35 all (53/620): mlockall.sh 20190515 01:45:50 all (54/620): datamove3.sh freework leak: 43/64. 20190515 01:45:54 all (55/620): msdos7.sh mount leak: 1/123 20190515 01:45:58 all (56/620): sndstat.sh 20190515 01:46:02 all (57/620): tmpfs14.sh mount leak: 1/124 20190515 01:46:16 all (58/620): vm_fault_dontneed.sh 20190515 01:46:24 all (59/620): rename6.sh mount leak: 1/125 20190515 01:46:32 all (60/620): tmpfs5.sh mount leak: 1/126 20190515 01:46:47 all (61/620): maxproc.sh witness_lock_list_get: witness exhausted maxproc limit exceeded by uid 65534 (pid 44255); see tuning(7) and login.conf(5) 20190515 01:46:55 all (62/620): seekdir.sh mount leak: 1/127 20190515 01:46:59 all (63/620): overcommit2.sh swap_pager: cannot allocate bio May 15 01:47:15 x4 kernel: pid 46682 (dd), uid 0 inumber 5 on /mnt: filesystem full mount leak: 1/128 20190515 01:47:19 all (64/620): ufssuspend.sh 20190515 01:47:29 all (65/620): contigmalloc2.sh 20190515 01:47:52 all (66/620): datamove5.sh mount leak: 1/129 20190515 01:48:11 all (67/620): contigmalloc3.sh 20190515 01:48:16 all (68/620): ffs_blkfree.sh newblk leak: 64957/65604. mount leak: 11/140 20190515 01:48:37 all (69/620): datamove4.sh mount leak: 1/141 20190515 01:48:47 all (70/620): forkbomb.sh 20190515 01:48:51 all (71/620): mkfifo6.sh mount leak: 1/142 20190515 01:48:58 all (72/620): sendmsg2.sh 20190515 01:49:08 all (73/620): rename8.sh mount leak: 1/143 20190515 01:49:42 all (74/620): tmpfs18.sh mount leak: 1/144 20190515 01:49:46 all (75/620): pread.sh mount leak: 3/147 20190515 01:50:18 all (76/620): socketpair3.sh 20190515 01:51:22 all (77/620): crossmp8.sh mount leak: 1/148 20190515 01:52:29 all (78/620): credleak.sh mount leak: 160/308 20190515 01:53:32 all (79/620): vmstat.sh 20190515 01:54:46 all (80/620): nfssillyrename.sh 20190515 01:54:51 all (81/620): tmpfs19.sh mount leak: 1/309 20190515 01:56:05 all (82/620): tmpfs8.sh mount leak: 1/310 20190515 01:56:56 all (83/620): tmpfs10.sh mount leak: 1/311 20190515 01:57:10 all (84/620): sendfile3.sh freework leak: 166/230. newblk leak: 12126/77730. 20190515 01:58:46 all (85/620): rename11.sh mount leak: 1/312 20190515 01:59:29 all (86/620): kevent6.sh mount leak: 2/314 20190515 02:00:30 all (87/620): overflow3.sh 20190515 02:02:33 all (88/620): tmpfs13.sh mount leak: 1/315 20190515 02:04:41 all (89/620): mmap27.sh 20190515 02:06:47 all (90/620): racct.sh 20190515 02:09:48 all (91/620): sendfile5.sh mount leak: 2/317 20190515 02:11:29 all (92/620): graid1_3.sh 20190515 02:13:41 all (93/620): mmap5.sh 20190515 02:15:09 all (94/620): nfs_halfpage2.sh 20190515 02:16:15 all (95/620): tmpfs16.sh mount leak: 1/318 20190515 02:19:19 all (96/620): msdos5.sh GEOM leak: 19/272 20190515 02:21:44 all (97/620): thr3.sh 20190515 02:24:54 all (98/620): fdatasync.sh 20190515 02:28:03 all (99/620): mlockall7.sh 20190515 02:31:10 all (100/620): contigmalloc.sh 20190515 02:34:27 all (101/620): ffs_sync.sh mount leak: 1/319 20190515 02:37:33 all (102/620): f_offset.sh 20190515 02:38:47 all (103/620): mmap14.sh 20190515 02:42:45 all (104/620): mountro.sh mount leak: 1/320 20190515 02:46:51 all (105/620): mountro5.sh May 15 02:47:47 x4 kernel: pid 49549 (swap), jid 0, uid 0, was killed: out of swap space May 15 02:48:24 x4 kernel: pid 49571 (swap), jid 0, uid 0, was killed: out of swap space May 15 02:48:34 x4 kernel: pid 49563 (swap), jid 0, uid 0, was killed: out of swap space mount leak: 1/321 20190515 02:49:03 all (106/620): mountu.sh mount leak: 3/324 20190515 02:49:36 all (107/620): umountf3.sh mount leak: 1/325 20190515 02:53:50 all (108/620): truncate8.sh mount leak: 2/327 20190515 02:58:43 all (109/620): umountf4.sh mount leak: 1/328 20190515 03:00:39 all (110/620): devfs.sh 20190515 03:05:45 all (111/620): nullfs.sh 20190515 03:10:50 all (112/620): pipe.sh 20190515 03:16:01 all (113/620): sendfile17.sh Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0xdeadc0ee fault code = supervisor read data, page not present instruction pointer = 0x20:0x1089740 stack pointer = 0x28:0x21216a3c frame pointer = 0x28:0x21216a50 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 65852 (sendfile17) trap number = 12 panic: page fault cpuid = 3 time = 1557883251 KDB: stack backtrace: db_trace_self_wrapper(e8ac4d,1c11188,0,2121684c,be79c1,...) at db_trace_self_wrapper+0x2a/frame 0x21216820 kdb_backtrace(7,3,3,212169fc,212169fc,...) at kdb_backtrace+0x2e/frame 0x21216880 vpanic(15bbd75,212168c4,212168c4,212168f8,1558676,...) at vpanic+0x121/frame 0x212168a4 panic(15bbd75,1649719,2db5000,0,deadc0ee,...) at panic+0x14/frame 0x212168b8 trap_fatal(13029a5,0,4,16480b0,332,...) at trap_fatal+0x356/frame 0x212168f8 trap_pfault(deadc0ee) at trap_pfault+0x51/frame 0x21216934 trap(212169fc,8,28,28,24582d80,...) at trap+0x3c0/frame 0x212169f0 calltrap() at 0xffc0316d/frame 0x212169f0 --- trap 0xc, eip = 0x1089740, esp = 0x21216a3c, ebp = 0x21216a50 --- unp_dispose(21995618) at unp_dispose+0x80/frame 0x21216a50 sofree(21995618) at sofree+0x293/frame 0x21216a7c soclose(21995618) at soclose+0x2f7/frame 0x21216ab4 soo_close(189eddc8,99d8a80) at soo_close+0x1f/frame 0x21216ac0 _fdrop(189eddc8,99d8a80) at _fdrop+0x18/frame 0x21216ad8 closef(189eddc8,99d8a80,189eddc8,99d8a80,21cb56f0,...) at closef+0x1e0/frame 0x21216b2c fdescfree_fds(1) at fdescfree_fds+0x98/frame 0x21216b54 fdescfree(99d8a80) at fdescfree+0x389/frame 0x21216bc4 exit1(99d8a80,0,0,21216cdc,1558d79,...) at exit1+0x47a/frame 0x21216bf4 sys_sys_exit(99d8a80,99d8d08) at sys_sys_exit+0x12/frame 0x21216c08 syscall(21216ce8,3b,3b,3b,ffbfe808,...) at syscall+0x2d9/frame 0x21216cdc Xint0x80_syscall() at 0xffc033b7/frame 0x21216cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x20555473, esp = 0xffbfe544, ebp = 0xffbfe6e0 --- __bss_end() at 0x20555473 KDB: enter: panic [ thread pid 65852 tid 100109 ] Stopped at kdb_enter+0x35: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 65852 tid 100109 td 0x99d8a80 kdb_enter(167ade6,167ade6) at kdb_enter+0x35/frame 0x21216878 vpanic(15bbd75,212168c4,212168c4,212168f8,1558676,...) at vpanic+0x139/frame 0x212168a4 panic(15bbd75,1649719,2db5000,0,deadc0ee,...) at panic+0x14/frame 0x212168b8 trap_fatal(13029a5,0,4,16480b0,332,...) at trap_fatal+0x356/frame 0x212168f8 trap_pfault(deadc0ee) at trap_pfault+0x51/frame 0x21216934 trap(212169fc,8,28,28,24582d80,...) at trap+0x3c0/frame 0x212169f0 calltrap() at 0xffc0316d/frame 0x212169f0 --- trap 0xc, eip = 0x1089740, esp = 0x21216a3c, ebp = 0x21216a50 --- unp_dispose(21995618) at unp_dispose+0x80/frame 0x21216a50 sofree(21995618) at sofree+0x293/frame 0x21216a7c soclose(21995618) at soclose+0x2f7/frame 0x21216ab4 soo_close(189eddc8,99d8a80) at soo_close+0x1f/frame 0x21216ac0 _fdrop(189eddc8,99d8a80) at _fdrop+0x18/frame 0x21216ad8 closef(189eddc8,99d8a80,189eddc8,99d8a80,21cb56f0,...) at closef+0x1e0/frame 0x21216b2c fdescfree_fds(1) at fdescfree_fds+0x98/frame 0x21216b54 fdescfree(99d8a80) at fdescfree+0x389/frame 0x21216bc4 exit1(99d8a80,0,0,21216cdc,1558d79,...) at exit1+0x47a/frame 0x21216bf4 sys_sys_exit(99d8a80,99d8d08) at sys_sys_exit+0x12/frame 0x21216c08 syscall(21216ce8,3b,3b,3b,ffbfe808,...) at syscall+0x2d9/frame 0x21216cdc Xint0x80_syscall() at 0xffc033b7/frame 0x21216cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x20555473, esp = 0xffbfe544, ebp = 0xffbfe6e0 --- __bss_end() at 0x20555473 db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x884400 curthread = 0x2a815000: pid 65821 tid 102699 "sendfile17" curpcb = 0x36a75d40 fpcurthread = 0x2a815000: pid 65821 "sendfile17" idlethread = 0x99d7700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x36a75d30 common_tssp = 0xffc01260 tlb gen = 140124631 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x7bee400 curthread = 0x94c8380: pid 21 tid 100097 "dom0" curpcb = 0xb3ded40 fpcurthread = none idlethread = 0x99d7380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0xb3ded30 common_tssp = 0xffc012c8 tlb gen = 140124631 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x7bf3400 curthread = 0x210ae380: pid 65860 tid 100111 "sendfile17" curpcb = 0x2122cd40 fpcurthread = 0x210ae380: pid 65860 "sendfile17" idlethread = 0x99d7000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x2122cd30 common_tssp = 0xffc01330 tlb gen = 140124629 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x7bf8400 curthread = 0x99d8a80: pid 65852 tid 100109 "sendfile17" curpcb = 0x21216d40 fpcurthread = 0x99d8a80: pid 65852 "sendfile17" idlethread = 0x99d6a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x21216d30 common_tssp = 0xffc01398 tlb gen = 140124631 curvnet = 0x9963040 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0x42d4f16c: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags (VI_ACTIVE) v_object 0x2ddeaea0 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfd1d11 at lockmgr_slock_hard+0x2c1 #1 0xfd22d6 at __lockmgr_args+0x126 #2 0x12c339f at ffs_lock+0x6f #3 0x1588948 at VOP_LOCK1_APV+0x58 #4 0x10c4c98 at _vn_lock+0x58 #5 0x10c2013 at kern_getdirentries+0x143 #6 0x10c22d5 at sys_getdirentries+0x25 #7 0x1558d79 at syscall+0x2d9 #8 0xffc033b7 at _end+0xfdc168c7 nlink=2, effnlink=2, size=512, extsize 0 generation=5a3d79b8, uid=1001, gid=0, flags=0x0 ino 1206610, on dev ada0s1e vnode 0x43064e38: tag ufs, type VREG usecount 7, writecount 0, refcount 8 flags (VI_ACTIVE) v_object 0x180dd000 ref 1 pages 1 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xfd1d11 at lockmgr_slock_hard+0x2c1 #1 0xfd22d6 at __lockmgr_args+0x126 #2 0x12c339f at ffs_lock+0x6f #3 0x1588948 at VOP_LOCK1_APV+0x58 #4 0x10c4c98 at _vn_lock+0x58 #5 0x10c3cea at vn_statfile+0x2a #6 0xfab197 at kern_fstat+0x77 #7 0xfab28e at sys_fstat+0x1e #8 0x1558d79 at syscall+0x2d9 #9 0xffc033b7 at _end+0xfdc168c7 nlink=1, effnlink=1, size=4096, extsize 0 generation=a604a43a, uid=0, gid=0, flags=0x0 ino 4, on dev md10a db:1:pho1> show allchains chain 1: thread 100129 (pid 65851, sh) sleeping on 0x255d449c "swread" chain 2: thread 100180 (pid 65822, sendfile17) sleeping on 0x218ae9fc "wait" chain 3: thread 101091 (pid 65820, sendfile17) sleeping on 0x2c0559fc "wait" chain 4: thread 102732 (pid 65811, sendfile17) sleeping on 0x2a8c4354 "wait" chain 5: thread 103457 (pid 65810, sendfile17) sleeping on 0x2ade8000 "wait" chain 6: thread 100247 (pid 64226, sh) sleeping on 0x256a8000 "wait" chain 7: thread 106423 (pid 36003, swap) sleeping on 0x4387ad24 "swread" chain 8: thread 100343 (pid 36000, swap) sleeping on 0x2f19eb50 "swread" chain 9: thread 101067 (pid 35999, swap) sleeping on 0x23ad15d4 "swread" chain 10: thread 100114 (pid 35997, swap) sleeping on 0x24df0364 "swread" chain 11: thread 102664 (pid 35996, swap) sleeping on 0x309515d4 "swread" chain 12: thread 101297 (pid 35995, swap) sleeping on 0x95fbd24 "swread" chain 13: thread 103892 (pid 35993, swap) sleeping on 0x2b46f9fc "wait" chain 14: thread 100189 (pid 35735, swap) sleeping on 0x1f05ed4 "nanslp" chain 15: thread 100261 (pid 35730, md10) sleeping on 0x186cd000 "mdwait" chain 16: thread 103490 (pid 33514, sh) sleeping on 0x2ae329fc "wait" chain 17: thread 100214 (pid 33513, tee) sleeping on 0x2dc35000 "piperd" chain 18: thread 100930 (pid 33512, sh) sleeping on 0x2ba0d000 "wait" chain 19: thread 100211 (pid 5402, cat) sleeping on 0x42befd40 "piperd" chain 20: thread 103124 (pid 5325, sh) sleeping on 0x2a967354 "wait" chain 21: thread 102248 (pid 5324, find) sleeping on 0x9683104 "biord" chain 22: thread 100259 (pid 5197, sh) sleeping on 0x256ce6a8 "wait" chain 23: thread 102731 (pid 5157, mail) sleeping on 0x17f1e9f0 "piperd" chain 24: thread 101517 (pid 5156, sh) sleeping on 0x30eea9fc "wait" chain 25: thread 103891 (pid 4957, sh) sleeping on 0x2b470000 "wait" chain 26: thread 102003 (pid 4921, lockf) sleeping on 0x31164354 "wait" chain 27: thread 102276 (pid 4821, sh) sleeping on 0x29e139fc "wait" chain 28: thread 100184 (pid 4792, sh) sleeping on 0x21d6f6a8 "wait" chain 29: thread 101296 (pid 920, mail) sleeping on 0x23b91b98 "piperd" chain 30: thread 100281 (pid 918, sh) sleeping on 0x256e36a8 "wait" chain 31: thread 102733 (pid 696, sh) sleeping on 0x2a8c4000 "wait" chain 32: thread 100262 (pid 683, lockf) sleeping on 0x256cd9fc "wait" chain 33: thread 100105 (pid 637, sh) sleeping on 0x17dfc6a8 "wait" chain 34: thread 103022 (pid 635, cron) sleeping on 0x21ba96a0 "piperd" chain 35: thread 100187 (pid 88683, sh) sleeping on 0x23b91000 "piperd" chain 36: thread 100358 (pid 88681, awk) sleeping on 0x2109a4f8 "piperd" chain 37: thread 100305 (pid 88611, top) sleeping on 0x17af9b64 "select" chain 38: thread 100356 (pid 88609, tail) sleeping on 0x251d3b00 "kqread" chain 39: thread 100316 (pid 88607, sh) sleeping on 0x257179fc "wait" chain 40: thread 100245 (pid 88606, sshd) sleeping on 0x24244da4 "select" chain 41: thread 100282 (pid 88605, sshd) sleeping on 0x242442a4 "select" chain 42: thread 100284 (pid 88604, sshd) sleeping on 0x17af8324 "select" chain 43: thread 100345 (pid 88598, sshd) sleeping on 0x251f0524 "select" chain 44: thread 100313 (pid 88597, sshd) sleeping on 0x17ad8be4 "select" chain 45: thread 100277 (pid 88596, sshd) sleeping on 0x17af8724 "select" chain 46: thread 100275 (pid 74969, sh) sleeping on 0x256e5000 "wait" chain 47: thread 100276 (pid 1473, sh) sleeping on 0x2109d000 "piperd" chain 48: thread 100265 (pid 1471, awk) sleeping on 0x21cd74f8 "piperd" chain 49: thread 100271 (pid 1470, sh) sleeping on 0x256a86a8 "wait" chain 50: thread 100273 (pid 1449, sh) sleeping on 0x256e56a8 "wait" chain 51: thread 100107 (pid 906, bash) sleeping on 0x17dfc354 "wait" chain 52: thread 100197 (pid 904, csh) sleeping on 0x222dc3b8 "pause" chain 53: thread 100185 (pid 903, su) sleeping on 0x21d6f354 "wait" chain 54: thread 100127 (pid 775, bash) sleeping on 0x185d86a8 "wait" chain 55: thread 100135 (pid 774, sshd) sleeping on 0x17ad90e4 "select" chain 56: thread 100178 (pid 772, sshd) sleeping on 0x17ad9364 "select" chain 57: thread 100204 (pid 770, getty) sleeping on 0x17c8e074 "ttyin" chain 58: thread 100203 (pid 769, getty) sleeping on 0x186d9c74 "ttyin" chain 59: thread 100202 (pid 768, getty) sleeping on 0x186da074 "ttyin" chain 60: thread 100174 (pid 767, getty) sleeping on 0x186da474 "ttyin" chain 61: thread 100181 (pid 766, getty) sleeping on 0x186da874 "ttyin" chain 62: thread 100112 (pid 765, getty) sleeping on 0x186dac74 "ttyin" chain 63: thread 100188 (pid 764, getty) sleeping on 0x186db074 "ttyin" chain 64: thread 100196 (pid 763, getty) sleeping on 0x186db474 "ttyin" chain 65: thread 100198 (pid 762, getty) sleeping on 0x186db874 "ttyin" chain 66: thread 100183 (pid 745, inetd) sleeping on 0x185e63a4 "select" chain 67: thread 100190 (pid 711, nfscbd: master) sleeping on 0x21da9ad0 "rpcsvc" chain 68: thread 100192 (pid 711, nfscbd: service) sleeping on 0x180c2f50 "rpcsvc" chain 69: thread 100193 (pid 711, nfscbd: service) sleeping on 0x180c2ed0 "rpcsvc" chain 70: thread 100194 (pid 711, nfscbd: service) sleeping on 0x180c3050 "rpcsvc" chain 71: thread 100191 (pid 710, nfscbd) sleeping on 0x21b8bcf8 "accept" chain 72: thread 100110 (pid 674, cron) sleeping on 0x1f05ed5 "nanslp" chain 73: thread 100182 (pid 670, sendmail) sleeping on 0x21d70064 "pause" chain 74: thread 100128 (pid 667, sendmail) sleeping on 0x17ad9624 "select" chain 75: thread 100108 (pid 664, sshd) sleeping on 0x17ad94a4 "select" chain 76: thread 100175 (pid 624, smartd) sleeping on 0x1f05ed6 "nanslp" chain 77: thread 100177 (pid 600, powerd) sleeping on 0x942b4e4 "select" chain 78: thread 100172 (pid 597, ntpd) sleeping on 0x17ad9564 "select" chain 79: thread 100121 (pid 559, NLM: master) sleeping on 0x180c21d0 "rpcsvc" chain 80: thread 100171 (pid 556, rpc.statd) sleeping on 0x17ad92e4 "select" chain 81: thread 100139 (pid 555, nfsd: master) sleeping on 0x180c2dd0 "rpcsvc" chain 82: thread 100140 (pid 555, nfsd: service) sleeping on 0x180c30d0 "rpcsvc" chain 83: thread 100141 (pid 555, nfsd: service) sleeping on 0x180c29d0 "rpcsvc" chain 84: thread 100142 (pid 555, nfsd: service) sleeping on 0x180c2bd0 "rpcsvc" chain 85: thread 100143 (pid 555, nfsd: service) sleeping on 0x180c2950 "rpcsvc" chain 86: thread 100144 (pid 555, nfsd: service) sleeping on 0x180c28d0 "rpcsvc" chain 87: thread 100145 (pid 555, nfsd: service) sleeping on 0x180c2850 "rpcsvc" chain 88: thread 100146 (pid 555, nfsd: service) sleeping on 0x180c27d0 "rpcsvc" chain 89: thread 100147 (pid 555, nfsd: service) sleeping on 0x21dab1d0 "rpcsvc" chain 90: thread 100148 (pid 555, nfsd: service) sleeping on 0x180c2750 "rpcsvc" chain 91: thread 100149 (pid 555, nfsd: service) sleeping on 0x180c26d0 "rpcsvc" chain 92: thread 100150 (pid 555, nfsd: service) sleeping on 0x180c2650 "rpcsvc" chain 93: thread 100151 (pid 555, nfsd: service) sleeping on 0x21dab150 "rpcsvc" chain 94: thread 100152 (pid 555, nfsd: service) sleeping on 0x180c25d0 "rpcsvc" chain 95: thread 100153 (pid 555, nfsd: service) sleeping on 0x21dab0d0 "rpcsvc" chain 96: thread 100154 (pid 555, nfsd: service) sleeping on 0x180c2550 "rpcsvc" chain 97: thread 100155 (pid 555, nfsd: service) sleeping on 0x180c24d0 "rpcsvc" chain 98: thread 100156 (pid 555, nfsd: service) sleeping on 0x21dab050 "rpcsvc" chain 99: thread 100157 (pid 555, nfsd: service) sleeping on 0x21da9f50 "rpcsvc" chain 100: thread 100158 (pid 555, nfsd: service) sleeping on 0x21da9ed0 "rpcsvc" chain 101: thread 100159 (pid 555, nfsd: service) sleeping on 0x180c2450 "rpcsvc" chain 102: thread 100160 (pid 555, nfsd: service) sleeping on 0x21da9e50 "rpcsvc" chain 103: thread 100161 (pid 555, nfsd: service) sleeping on 0x21da9dd0 "rpcsvc" chain 104: thread 100162 (pid 555, nfsd: service) sleeping on 0x21da9d50 "rpcsvc" chain 105: thread 100163 (pid 555, nfsd: service) sleeping on 0x180c23d0 "rpcsvc" chain 106: thread 100164 (pid 555, nfsd: service) sleeping on 0x180c2350 "rpcsvc" chain 107: thread 100165 (pid 555, nfsd: service) sleeping on 0x21da9cd0 "rpcsvc" chain 108: thread 100166 (pid 555, nfsd: service) sleeping on 0x21da9c50 "rpcsvc" chain 109: thread 100167 (pid 555, nfsd: service) sleeping on 0x180c22d0 "rpcsvc" chain 110: thread 100168 (pid 555, nfsd: service) sleeping on 0x21da9bd0 "rpcsvc" chain 111: thread 100169 (pid 555, nfsd: service) sleeping on 0x180c2250 "rpcsvc" chain 112: thread 100170 (pid 555, nfsd: service) sleeping on 0x21da9b50 "rpcsvc" chain 113: thread 100106 (pid 552, nfsd) sleeping on 0x17ad9524 "select" chain 114: thread 100136 (pid 546, mountd) sleeping on 0x185e6864 "select" chain 115: thread 100134 (pid 539, nfsuserd) sleeping on 0x185e6a24 "select" chain 116: thread 100133 (pid 538, nfsuserd) sleeping on 0x185e6824 "select" chain 117: thread 100132 (pid 537, nfsuserd) sleeping on 0x17ad94e4 "select" chain 118: thread 100130 (pid 536, nfsuserd) sleeping on 0x185e69e4 "select" chain 119: thread 100131 (pid 535, nfsuserd) sleeping on 0x99cb70c "pause" chain 120: thread 100126 (pid 492, watchdogd) sleeping on 0x1f05ed7 "nanslp" chain 121: thread 100122 (pid 407, rpcbind) sleeping on 0x17ad95e4 "select" chain 122: thread 100125 (pid 398, syslogd) sleeping on 0x942b5a4 "select" chain 123: thread 100123 (pid 326, devd) sleeping on 0x17ad95a4 "select" chain 124: thread 100104 (pid 25, syncer) sleeping on 0x1f80624 "syncer" chain 125: thread 100099 (pid 23, bufdaemon) sleeping on 0x1f7ffbc "qsleep" chain 126: thread 100102 (pid 23, bufspacedaemon-0) sleeping on 0x1c344c0 "-" chain 127: thread 100115 (pid 23, /home worker) sleeping on 0x17a07484 "sdflush" chain 128: thread 100116 (pid 23, /tmp worker) sleeping on 0x17a07284 "sdflush" chain 129: thread 100117 (pid 23, /usr worker) sleeping on 0x17a07084 "sdflush" chain 130: thread 100118 (pid 23, /var worker) sleeping on 0x17a06e84 "sdflush" chain 131: thread 100515 (pid 23, /mnt worker) sleeping on 0x2d3cd884 "sdflush" chain 132: thread 100098 (pid 22, vmdaemon) sleeping on 0x1f95a78 "psleep" chain 133: thread 100100 (pid 21, laundry: dom0) sleeping on 0x1f95590 "swbufa" chain 134: thread 100101 (pid 21, uma) sleeping on 0x12de610 "umarcl" chain 135: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1d6b4c4 "-" chain 136: thread 100090 (pid 19, sctp_iterator) sleeping on 0x1fc90fc "waiting_for_work" chain 137: thread 100089 (pid 18, soaiod4) sleeping on 0x1f7fbe8 "-" chain 138: thread 100088 (pid 17, soaiod3) sleeping on 0x1f7fbe8 "-" chain 139: thread 100087 (pid 16, soaiod2) sleeping on 0x1f7fbe8 "-" chain 140: thread 100086 (pid 9, soaiod1) sleeping on 0x1f7fbe8 "-" chain 141: thread 100083 (pid 8, fdc0) sleeping on 0x17b2483c "-" chain 142: thread 100048 (pid 15, usbus0) sleeping on 0x17b5eacc "-" chain 143: thread 100049 (pid 15, usbus0) sleeping on 0x17b5eafc "-" chain 144: thread 100050 (pid 15, usbus0) sleeping on 0x17b5eb2c "-" chain 145: thread 100051 (pid 15, usbus0) sleeping on 0x17b5eb5c "-" chain 146: thread 100052 (pid 15, usbus0) sleeping on 0x17b5eb8c "-" chain 147: thread 100054 (pid 15, usbus1) sleeping on 0x17e15acc "-" chain 148: thread 100055 (pid 15, usbus1) sleeping on 0x17e15afc "-" chain 149: thread 100056 (pid 15, usbus1) sleeping on 0x17e15b2c "-" chain 150: thread 100057 (pid 15, usbus1) sleeping on 0x17e15b5c "-" chain 151: thread 100058 (pid 15, usbus1) sleeping on 0x17e15b8c "-" chain 152: thread 100060 (pid 15, usbus2) sleeping on 0x17e1cacc "-" chain 153: thread 100061 (pid 15, usbus2) sleeping on 0x17e1cafc "-" chain 154: thread 100062 (pid 15, usbus2) sleeping on 0x17e1cb2c "-" chain 155: thread 100063 (pid 15, usbus2) sleeping on 0x17e1cb5c "-" chain 156: thread 100064 (pid 15, usbus2) sleeping on 0x17e1cb8c "-" chain 157: thread 100065 (pid 15, usbus3) sleeping on 0x17e20acc "-" chain 158: thread 100066 (pid 15, usbus3) sleeping on 0x17e20afc "-" chain 159: thread 100067 (pid 15, usbus3) sleeping on 0x17e20b2c "-" chain 160: thread 100068 (pid 15, usbus3) sleeping on 0x17e20b5c "-" chain 161: thread 100069 (pid 15, usbus3) sleeping on 0x17e20b8c "-" chain 162: thread 100070 (pid 15, usbus4) sleeping on 0x17e26acc "-" chain 163: thread 100071 (pid 15, usbus4) sleeping on 0x17e26afc "-" chain 164: thread 100072 (pid 15, usbus4) sleeping on 0x17e26b2c "-" chain 165: thread 100073 (pid 15, usbus4) sleeping on 0x17e26b5c "-" chain 166: thread 100074 (pid 15, usbus4) sleeping on 0x17e26b8c "-" chain 167: thread 100076 (pid 15, usbus5) sleeping on 0x17b49ca4 "-" chain 168: thread 100077 (pid 15, usbus5) sleeping on 0x17b49cd4 "-" chain 169: thread 100078 (pid 15, usbus5) sleeping on 0x17b49d04 "-" chain 170: thread 100079 (pid 15, usbus5) sleeping on 0x17b49d34 "-" chain 171: thread 100080 (pid 15, usbus5) sleeping on 0x17b49d64 "-" chain 172: thread 100043 (pid 7, doneq0) sleeping on 0x1ca48c0 "-" chain 173: thread 100096 (pid 7, scanner) sleeping on 0x1ca47ec "-" chain 174: thread 100042 (pid 14, sequencer 00) sleeping on 0x17a0824c "seqstate" chain 175: thread 100041 (pid 6, crypto returns 3) sleeping on 0x95fe0c4 "crypto_ret_wait" chain 176: thread 100040 (pid 5, crypto returns 2) sleeping on 0x95fe08c "crypto_ret_wait" chain 177: thread 100039 (pid 4, crypto returns 1) sleeping on 0x95fe054 "crypto_ret_wait" chain 178: thread 100038 (pid 3, crypto returns 0) sleeping on 0x95fe01c "crypto_ret_wait" chain 179: thread 100037 (pid 2, crypto) sleeping on 0x1f94830 "crypto_wait" chain 180: thread 100030 (pid 13, g_event) sleeping on 0x1fb90bc "-" chain 181: thread 100031 (pid 13, g_up) sleeping on 0x1fb90c4 "-" chain 182: thread 100032 (pid 13, g_down) sleeping on 0x1fb90b8 "-" chain 183: thread 100002 (pid 1, init) sleeping on 0x99ca9fc "wait" chain 184: thread 100001 (pid 10, audit) sleeping on 0x1fc98e0 "audit_worker_cv" chain 185: thread 100000 (pid 0, swapper) sleeping on 0x1fb90dc "swapin" chain 186: thread 100009 (pid 0, thread taskq) sleeping on 0x99fee80 "-" chain 187: thread 100011 (pid 0, config_0) sleeping on 0x9432800 "-" chain 188: thread 100012 (pid 0, kqueue_ctx taskq) sleeping on 0x99fed80 "-" chain 189: thread 100013 (pid 0, aiod_kick taskq) sleeping on 0x99fed00 "-" chain 190: thread 100014 (pid 0, if_io_tqg_0) sleeping on 0x9432780 "-" chain 191: thread 100015 (pid 0, if_io_tqg_1) sleeping on 0x9432700 "-" chain 192: thread 100016 (pid 0, if_io_tqg_2) sleeping on 0x9432680 "-" chain 193: thread 100017 (pid 0, if_io_tqg_3) sleeping on 0x9432600 "-" chain 194: thread 100018 (pid 0, softirq_0) sleeping on 0x94db080 "-" chain 195: thread 100019 (pid 0, softirq_1) sleeping on 0x94db000 "-" chain 196: thread 100020 (pid 0, softirq_2) sleeping on 0x94daf00 "-" chain 197: thread 100021 (pid 0, softirq_3) sleeping on 0x94dae80 "-" chain 198: thread 100022 (pid 0, if_config_tqg_0) sleeping on 0x94dae00 "-" chain 199: thread 100029 (pid 0, firmware taskq) sleeping on 0x958d080 "-" chain 200: thread 100033 (pid 0, crypto_0) sleeping on 0x958d000 "-" chain 201: thread 100034 (pid 0, crypto_1) sleeping on 0x958d000 "-" chain 202: thread 100035 (pid 0, crypto_2) sleeping on 0x958d000 "-" chain 203: thread 100036 (pid 0, crypto_3) sleeping on 0x958d000 "-" chain 204: thread 100085 (pid 0, mca taskq) sleeping on 0x17db1d80 "-" chain 205: thread 100091 (pid 0, acpi_task_0) sleeping on 0x17db1d00 "-" chain 206: thread 100092 (pid 0, acpi_task_1) sleeping on 0x17db1d00 "-" chain 207: thread 100093 (pid 0, acpi_task_2) sleeping on 0x17db1d00 "-" chain 208: thread 100095 (pid 0, CAM taskq) sleeping on 0x958cf00 "-" db:1:pho1> show mount 0x189c6000 /dev/ada0s1a on / (ufs) 0x188a5000 devfs on /dev (devfs) 0x188a4000 /dev/ada0s1f on /home (ufs) 0x188a3000 /dev/label/tmp on /tmp (ufs) 0x215b0000 /dev/ada0s1d on /usr (ufs) 0x215af000 /dev/ada0s1e on /var (ufs) 0x215ae000 procfs on /proc (procfs) 0x29ad2000 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 0 Buf domain 0 freebufs 462 lofreebufs 311 hifreebufs 466 bufspace 33611776 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 13 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5256 (33085440) dirtyq count 13 (212992) wakeup 0 lim 28 CPU 8, 25, 6, 7, Locked buffers: 2 space 18432 Total buffers: 5779 space 33611776 db:1:pho1> show page vm_cnt.v_free_count: 14504 vm_cnt.v_inactive_count: 1262 vm_cnt.v_active_count: 656278 vm_cnt.v_laundry_count: 69685 Loading /boot/loader.conf.local -- Test scenario: misc/sendfile17.sh