KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r343866: Thu Feb 7 12:02:35 CET 2019 pho@t2.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 11 boot pages configured startup_alloc from "UMA Kegs", 10 boot pages left startup_alloc from "UMA Zones", 9 boot pages left startup_alloc from "UMA Zones", 8 boot pages left startup_alloc from "UMA Hash", 7 boot pages left startup_alloc from "UMA Zones", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.24-MHz 686-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100000 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 3038281728 (2897 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20190108/tbfadt-850) ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard Launching APs: 1 16 21 6 19 9 5 14 3 7 4 17 2 11 20 18 22 13 8 12 15 23 10 Timecounter "TSC" frequency 1995237216 Hz quality 1000 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0x1658210, 0) error 19 kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pcib0: Ignoring 3 range above 4GB (0x380000000000-0x38007fffffff) pci0: on pcib0 pcib1: irq 47 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 47 at device 2.0 on pci0 pci2: on pcib2 pcib3: irq 47 at device 3.0 on pci0 pci3: on pcib3 pcib4: irq 16 at device 17.0 on pci0 pci4: on pcib4 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xd0420000-0xd04203ff irq 22 at device 26.0 on pci0 usbus0: waiting for BIOS to give up control usbus0: timed out waiting for BIOS usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 hdac0: mem 0xd7f20000-0xd7f23fff irq 22 at device 27.0 on pci0 pcib5: irq 16 at device 28.0 on pci0 pci5: on pcib5 igb0: port 0x1020-0x103f mem 0xd0220000-0xd023ffff,0xd0250000-0xd0253fff irq 16 at device 0.0 on pci5 igb0: PHY reset is blocked due to SOL/IDER session. igb0: Using 1024 tx descriptors and 1024 rx descriptors igb0: Using 8 rx queues 8 tx queues igb0: Using MSI-X interrupts with 9 vectors igb0: Ethernet address: 00:1e:67:56:b6:9b igb0: link state changed to UP igb1: port 0x1000-0x101f mem 0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 17 at device 0.1 on pci5 igb1: PHY reset is blocked due to SOL/IDER session. igb1: Using 1024 tx descriptors and 1024 rx descriptors igb1: Using 8 rx queues 8 tx queues igb1: Using MSI-X interrupts with 9 vectors igb1: Ethernet address: 00:1e:67:56:b6:9c pcib6: irq 17 at device 28.5 on pci0 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 pci7: at device 0.0 (no driver attached) pcib8: irq 18 at device 28.6 on pci0 pci8: on pcib8 xhci0: mem 0xd0010000-0xd001ffff,0xd0000000-0xd0001fff irq 18 at device 0.0 on pci8 xhci0: 64 bytes context size, 64-bit DMA usbus1 on xhci0 usbus1: 5.0Gbps Super Speed USB v3.0 pcib9: irq 19 at device 28.7 on pci0 pci9: on pcib9 ehci1: mem 0xd0410000-0xd04103ff irq 20 at device 29.0 on pci0 usbus2: waiting for BIOS to give up control usbus2: timed out waiting for BIOS usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 pcib10: at device 30.0 on pci0 pci10: on pcib10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x3070-0x3077,0x3060-0x3063,0x3050-0x3057,0x3040-0x3043,0x3020-0x303f mem 0xd0400000-0xd04007ff irq 21 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pcib11: on acpi0 pcib11: Ignoring 3 range above 4GB (0x380080000000-0x3800ffffffff) pci11: on pcib11 pcib12: at device 0.0 on pci11 pci12: on pcib12 pcib13: irq 71 at device 1.0 on pci11 pci13: on pcib13 pcib14: irq 71 at device 2.0 on pci11 pci14: on pcib14 vgapci0: port 0xa000-0xa0ff mem 0xe0000000-0xefffffff,0xd8020000-0xd803ffff irq 56 at device 0.0 on pci14 vgapci0: Boot video device hdac1: mem 0xd8040000-0xd8043fff irq 60 at device 0.1 on pci14 pcib15: irq 71 at device 3.0 on pci11 pci15: on pcib15 acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 pcib16: on acpi0 pci16: on pcib16 pci16: at device 14.1 (no driver attached) pci16: at device 19.1 (no driver attached) pci16: at device 19.4 (no driver attached) pci16: at device 19.5 (no driver attached) pcib17: on acpi0 pci17: on pcib17 pci17: at device 14.1 (no driver attached) pci17: at device 19.1 (no driver attached) pci17: at device 19.4 (no driver attached) pci17: at device 19.5 (no driver attached) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xcffff,0xda000-0xdafff,0xdb000-0xdbfff pnpid ORM0000 on isa0 ata0: at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata1: at port 0x170-0x177,0x376 irq 15 on isa0 ppc0: parallel port not found. est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21,23 and 24,26 on hdaa0 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device cd0 at ahcich1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number R93E68ED300PN5 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed pcm1: at nid 27 and 25 on hdaa0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number CVCV317607P0240CGN ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 228936MB (468862128 512 byte sectors) pcm2: at nid 30 on hdaa0 hdacc1: at cad 0 on hdac1 hdaa1: at nid 1 on hdacc1 pcm3: at nid 3 on hdaa1 Trying to mount root from ufs:/dev/gpt/ssdrootfs [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. ugen0.1: at usbus0 ugen2.1: at usbus2 uhub0: on usbus0 ugen1.1: <0x104c XHCI root HUB> at usbus1 uhub1: on usbus2 Expensive timeout(9) function: 0xf4bc50(0x1dbdda8) 0.006795580 s uhub2: <0x104c XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting ddb. warning: total configured swap (16777216 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/gpt/ssdrootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/gpt/ssdrootfs: clean, 25643267 free (31947 frags, 3201415 blocks, 0.1% fragmentation) Mounting local filesystems:. uhub2: 4 ports with 4 removable, self powered lock order reversal: 1st 0x30432b94 ufs (ufs) @ kern/vfs_subr.c:2590 2nd 0x8c0946c bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:282 3rd 0x310a2034 ufs (ufs) @ kern/vfs_subr.c:2590 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6ddb at lockmgr_xlock_hard+0x5b #3 0x10c7632 at __lockmgr_args+0x3d2 #4 0x14364f4 at ffs_lock+0x94 #5 0x1734b69 at VOP_LOCK1_APV+0x69 #6 0x11e2871 at _vn_lock+0x61 #7 0x11ce9ae at vget+0xae #8 0x11bfb5d at vfs_hash_get+0xcd #9 0x1431164 at ffs_vgetf+0x44 #10 0x14264a7 at softdep_sync_buf+0x9d7 #11 0x143732f at ffs_syncvnode+0x27f #12 0x1408d31 at ffs_truncate+0x9c1 #13 0x1440bed at ufs_direnter+0x62d #14 0x144a6a2 at ufs_makeinode+0x5a2 #15 0x14463f8 at ufs_create+0x38 #16 0x17319e5 at VOP_CREATE_APV+0x85 #17 0x11e20c0 at vn_open_cred+0x260 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat /usr/local/lib/e2fsprogs /usr/local/lib/mozjpeg /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/llvm60/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: t2.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: at usbus0 uhub3 on uhub0 uhub3: on usbus0 ugen2.2: at usbus2 uhub4 on uhub1 uhub4: on usbus2 . lo0: link state changed to UP uhub3: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 ugen2.3: at usbus2 uhub5 on uhub4 uhub5: on usbus2 uhub5: MTT enabled uhub5: 4 ports with 4 removable, self powered ugen2.4: at usbus2 ukbd1 on uhub4 ukbd1: on usbus2 kbd3 at ukbd1 Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8843 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9b inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:b69b%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseT ) status: active nd6 options=23 igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Configuring vt: keymap. ums0 on uhub3 ums0: on usbus0 ums1 on uhub4 ums1: on usbus2 ums0: 18 buttons and [XYZT] coordinates ID=2 ums1: 3 buttons and [Z] coordinates ID=0 Starting ums0 moused. Starting ums1 moused. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. Creating and/or trimming log files. Starting syslogd. Starting watchdogd. No core dumps found. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0x8c5d778 bufwait (bufwait) @ kern/vfs_bio.c:3906 2nd 0x3074dc00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10fe5d4 at _sx_xlock+0x64 #3 0x143c016 at ufsdirhash_add+0x56 #4 0x1440a43 at ufs_direnter+0x483 #5 0x144990f at ufs_mkdir+0x83f #6 0x1733e65 at VOP_MKDIR_APV+0x85 #7 0x11deff4 at kern_mkdirat+0x1d4 #8 0x11dee12 at sys_mkdir+0x32 #9 0x16fe941 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcd7fd3a3 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. sysctl: unknown oid 'debug.mnowait_failure.zalloc_whitelist' at line 13 sysctl: unknown oid 'debug.fail_point.uma_zalloc_arg' at line 14 Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Thu Feb 7 12:23:44 CET 2019 FreeBSD/i386 (t2.osted.lan) (ttyu1) login: Feb 7 12:27:07 t2 su[931]: pho to root on /dev/pts/0 20190207 12:29:10 all (1/2): marcus7.sh lock order reversal: 1st 0x32d0c5e4 ufs (ufs) @ kern/vfs_mount.c:1353 2nd 0x32d8de6c devfs (devfs) @ ufs/ffs/ffs_softdep.c:1914 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6959 at lockmgr_lock_fast_path+0x189 #3 0x11bbb1b at vop_stdlock+0x3b #4 0x1734b69 at VOP_LOCK1_APV+0x69 #5 0x11e2871 at _vn_lock+0x61 #6 0x1412b56 at softdep_flushworklist+0x66 #7 0x143488a at ffs_sync+0x4ba #8 0x11c684b at dounmount+0x53b #9 0x11c61e5 at sys_unmount+0x305 #10 0x16fe941 at syscall+0x361 #11 0xffc053a7 at ums_devclass+0xcd7fd3a3 mount leak: 2/35 20190207 12:39:33 all (2/2): md2.sh lock order reversal: 1st 0x3a3c9d1c ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:118 2nd 0x30459750 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1685 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6ddb at lockmgr_xlock_hard+0x5b #3 0x10c7632 at __lockmgr_args+0x3d2 #4 0x1431278 at ffs_vgetf+0x158 #5 0x1431117 at ffs_vget+0x37 #6 0x1445991 at ufs_root+0x31 #7 0x143d111 at ufs_extattr_autostart+0x61 #8 0x1433adc at ffs_mount+0x202c #9 0x11c5285 at vfs_domount+0x805 #10 0x11c446d at vfs_donmount+0x84d #11 0x11c3bf2 at sys_nmount+0x72 #12 0x16fe941 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcd7fd3a3 Expensive timeout(9) function: 0x1458530(0) 0.021697744 s g_vfs_done():md10[WRITE(offset=165838848, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165871616, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165904384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165937152, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165969920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166002688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166035456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166068224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166100992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166133760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166166528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166199296, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166232064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165871616, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165904384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165937152, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165969920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166002688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166035456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166068224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166100992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166133760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166166528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166199296, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166232064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=165969920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166002688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166035456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166068224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166100992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166133760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166166528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166199296, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166232064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166100992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166133760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166166528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166199296, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166232064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166199296, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166232064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181829632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181862400, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=111935488, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168951808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181469184, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181501952, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181534720, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181567488, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181600256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181633024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181665792, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181698560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181731328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181764096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181796864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181829632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181862400, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168951808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181534720, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181567488, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181600256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181633024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181665792, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181698560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181731328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181764096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181796864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181829632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181862400, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168951808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181600256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181633024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181665792, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181698560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181731328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181764096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181796864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181829632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181862400, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=168951808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=181796864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106430464, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=69500928, length=8192)]error = 28 g_vfs_done():md10[WRITE(offset=82477056, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83034112, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187695104, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187858944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231407616, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231440384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231505920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=82477056, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83034112, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96370688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96403456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96436224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96468992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96501760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=104660992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187695104, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187858944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231440384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231505920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96370688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96403456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96436224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96468992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96501760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=104660992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187695104, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187858944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231505920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96370688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96403456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96436224, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96468992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96501760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187695104, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187858944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231505920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96468992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96501760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187695104, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187858944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96468992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=96501760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=231538688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187891712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=83066880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=69484544, length=8192)]error = 28 g_vfs_done():md10[WRITE(offset=69533696, length=8192)]error = 28 Expensive timeout(9) function: 0x132a810(0x363405c8) 0.056296117 s Expensive timeout(9) function: 0x132a810(0x3986a2e4) 0.161399864 s g_vfs_done():md10[WRITE(offset=12058624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=12091392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=178847744, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=12058624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=12091392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=178847744, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=2232320, length=12288)]error = 28 g_vfs_done():md10[WRITE(offset=237993984, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=238026752, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263520256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263553024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=78905344, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88965120, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88997888, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154075136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187072512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187105280, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187138048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187170816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=217841664, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=238026752, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263520256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263553024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=78905344, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88965120, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88997888, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154075136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187072512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187105280, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187138048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187170816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=217841664, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=238026752, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263520256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263553024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88965120, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88997888, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154075136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187072512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187105280, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187138048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187170816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=217841664, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263520256, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263553024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88997888, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154075136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187138048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187170816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=263553024, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154075136, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=187170816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81428480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91586560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91619328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=123011072, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=156565504, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192151552, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192184320, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192217088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192249856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=241434624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=241467392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=266436608, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81428480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=123011072, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=156565504, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192151552, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192184320, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192217088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192249856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81428480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91652096, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91684864, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=151453696, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=166920192, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=244416512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=38567936, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=38600704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91062272, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=38600704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91324416, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=112885760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=112918528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=141656064, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=214138880, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=244449280, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=244482048, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=244514816, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=244547584, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=38600704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=91324416, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=112885760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=46792704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=46792704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=46727168, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=46759936, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=46792704, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=174292992, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149585920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149749760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=178323456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59473920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59506688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59539456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149749760, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=178323456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59473920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59506688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59539456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59473920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59506688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59539456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59473920, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59506688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=59539456, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149782528, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=138682368, length=4096)]error = 28 g_vfs_done():md10[WRITE(offset=220200960, length=32768)]error = 28 pts leak: 1/2 mount leak: 2/37 20190207 13:00:54 all (1/2): marcus7.sh mount leak: 1/38 20190207 13:11:22 all (2/2): md2.sh g_vfs_done():md10[WRITE(offset=230195200, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=74252288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=78282752, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90243072, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=220364800, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=230293504, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=74317824, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=220364800, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=220364800, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=90341376, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=137035776, length=24576)]error = 28 g_vfs_done():md10[WRITE(offset=69533696, length=8192)]error = 28 g_vfs_done():md10[WRITE(offset=69500928, length=12288)]error = 28 g_vfs_done():md10[WRITE(offset=69566464, length=4096)]error = 28 g_vfs_done():md10[WRITE(offset=190414848, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=190447616, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=190480384, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192380928, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192413696, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192446464, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192479232, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192512000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192544768, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192577536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192610304, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149061632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192479232, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192512000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192544768, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192577536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192610304, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149061632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192512000, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192544768, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192577536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192610304, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149061632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192577536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192610304, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=149061632, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192577536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=192610304, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103710720, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106233856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106561536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108658688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103350272, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103383040, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103415808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103710720, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106561536, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103383040, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103415808, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=103710720, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108658688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106921984, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=106921984, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108658688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108658688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108658688, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107020288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=88440832, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108003328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154271744, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154304512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167313408, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167346176, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167378944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167411712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167444480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=84606976, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107970560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108003328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154304512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167313408, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167346176, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167378944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167411712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167444480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=84606976, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107970560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108003328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=154304512, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167346176, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167378944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167411712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167444480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=107970560, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=108003328, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167346176, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167378944, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167411712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167444480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167411712, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=167444480, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=81264640, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=71761920, length=24576)]error = 28 g_vfs_done():md10[WRITE(offset=71729152, length=24576)]error = 28 g_vfs_done():md10[WRITE(offset=71794688, length=20480)]error = 28 g_vfs_done():md10[WRITE(offset=71860224, length=12288)]error = 28 g_vfs_done():md10[WRITE(offset=141623296, length=16384)]error = 28 g_vfs_done():md10[WRITE(offset=71827456, length=20480)]error = 28 mount leak: 2/40 20190207 13:32:17 all: done 20190207 14:04:58 all (1/614): ptrace4.sh lock order reversal: (sleepable after non-sleepable) 1st 0x1969ce38 pfs_node (pfs_node) @ fs/pseudofs/pseudofs_internal.h:105 2nd 0x8fe4918 pidhash (pidhash) @ kern/kern_proc.c:386 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10ff8f4 at _sx_slock_int+0x64 #3 0x10d79b8 at pfind+0x78 #4 0x102a003 at pfs_readdir+0x163 #5 0x1734485 at VOP_READDIR_APV+0x85 #6 0x11df589 at kern_getdirentries+0x1f9 #7 0x11df826 at sys_getdirentries+0x46 #8 0x16fe941 at syscall+0x361 #9 0xffc053a7 at exit_tag+0xcd7f2377 mount leak: 1/41 20190207 14:09:12 all (2/614): crossmp7.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (13 allocations, 22592 bytes leaked). 20190207 14:11:47 all (3/614): crossmp9.sh 20190207 14:16:54 all (4/614): mlockall3.sh 20190207 14:19:59 all (5/614): suj12.sh GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. mount leak: 1/42 20190207 14:41:10 all (6/614): nfs15.sh lock order reversal: 1st 0x8c8e0cc bufwait (bufwait) @ fs/nfsclient/nfs_clvnops.c:2756 2nd 0x31332b94 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:285 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10fe5d4 at _sx_xlock+0x64 #3 0x118db94 at sblock+0x74 #4 0x1192c2e at sosend_generic+0xde #5 0x11931d3 at sosend+0x73 #6 0x13caf28 at clnt_vc_call+0x4a8 #7 0x13c96e7 at clnt_reconnect_call+0x597 #8 0xfaf610 at newnfs_request+0x8e0 #9 0xfef9ef at nfscl_request+0x8f #10 0xfd9475 at nfsrpc_commit+0x115 #11 0xfe1f66 at ncl_commit+0x196 #12 0xfe247b at ncl_flush+0x47b #13 0xfe5a9b at nfs_fsync+0x3b #14 0x11bd8fe at vop_sigdefer+0x6e #15 0xfe12d9 at nfs_vnodeops_bypass+0x19 #16 0x1733709 at VOP_FSYNC_APV+0x89 #17 0x11a80d7 at bufsync+0x47 lock order reversal: 1st 0x8d1a77c bufwait (bufwait) @ kern/vfs_bio.c:1660 2nd 0x218f580 cpslk (cpslk) @ i386/i386/copyout.c:121 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10fe5d4 at _sx_xlock+0x64 #3 0x16d5728 at cp_slow0+0x178 #4 0x16d5b1a at copyout+0xca #5 0x1159d33 at uiomove_faultflag+0x163 #6 0x1159bbf at uiomove+0x1f #7 0x11e35bb at vn_io_fault_uiomove+0x3b #8 0xff2071 at ncl_bioread+0xce1 #9 0xfe6ee6 at nfs_readdir+0x1c6 #10 0x11bd8fe at vop_sigdefer+0x6e #11 0xfe12d9 at nfs_vnodeops_bypass+0x19 #12 0x173448d at VOP_READDIR_APV+0x8d #13 0x11df589 at kern_getdirentries+0x1f9 #14 0x11df826 at sys_getdirentries+0x46 #15 0x16fe941 at syscall+0x361 #16 0xffc053a7 at exit_tag+0xcd7f2377 20190207 14:46:23 all (7/614): sendfile5.sh lock order reversal: 1st 0x314ac374 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:282 2nd 0x4728a750 ufs (ufs) @ kern/kern_sendfile.c:699 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6aa9 at lockmgr_slock_hard+0x79 #3 0x10c73a8 at __lockmgr_args+0x148 #4 0x14364f4 at ffs_lock+0x94 #5 0x1734b69 at VOP_LOCK1_APV+0x69 #6 0x11e2871 at _vn_lock+0x61 #7 0x10f1946 at vn_sendfile+0x816 #8 0x10f390b at sendfile+0x19b #9 0x10f3766 at sys_sendfile+0x16 #10 0x16fe941 at syscall+0x361 #11 0xffc053a7 at exit_tag+0xcd7f2377 freework leak: 9/17. newblk leak: 28662/28669. mount leak: 3/45 20190207 14:47:38 all (8/614): aesni.sh aesni0: on motherboard GEOM_ELI: Device md10.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: md10 has been killed. GEOM_ELI: Device md10.eli destroyed. aesni0: detached mount leak: 1/46 20190207 14:53:06 all (9/614): quota4.sh lock order reversal: 1st 0x45d13478 syncer (syncer) @ kern/vfs_subr.c:2170 2nd 0x3b1e75e4 ufs (ufs) @ kern/vfs_subr.c:2590 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6ddb at lockmgr_xlock_hard+0x5b #3 0x10c7632 at __lockmgr_args+0x3d2 #4 0x14364f4 at ffs_lock+0x94 #5 0x1734b69 at VOP_LOCK1_APV+0x69 #6 0x11e2871 at _vn_lock+0x61 #7 0x11ce9ae at vget+0xae #8 0x1444dad at qsync+0x6d #9 0x1434a3a at ffs_sync+0x66a #10 0x11d6709 at sync_fsync+0xf9 #11 0x1733701 at VOP_FSYNC_APV+0x81 #12 0x11d4186 at sched_sync+0x2b6 #13 0x10b117e at fork_exit+0x7e #14 0xffc053ba at exit_tag+0xcd7f238a lock order reversal: 1st 0x4845d548 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:757 2nd 0x46789000 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:763 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10d1b69 at __mtx_lock_flags+0x89 #3 0x144419d at quotaoff_inchange+0xbd #4 0x14442af at quotaoff+0xbf #5 0x1431064 at ffs_flushfiles+0x164 #6 0x1412dba at softdep_flushfiles+0x6a #7 0x1433d08 at ffs_unmount+0xe8 #8 0x11c6836 at dounmount+0x526 #9 0x11c61e5 at sys_unmount+0x305 #10 0x16fe941 at syscall+0x361 #11 0xffc053a7 at exit_tag+0xcd7f2377 newblk leak: 1/28670. mount leak: 1/47 20190207 14:54:12 all (10/614): audit.sh 20190207 14:54:16 all (11/614): namecache2.sh mount leak: 1/48 20190207 14:55:46 all (12/614): snap5.sh 20190207 14:55:51 all (13/614): marcus2.sh mount leak: 1/49 20190207 15:17:24 all (14/614): symlink3.sh mount leak: 10/59 20190207 15:37:41 all (15/614): umount.sh mount leak: 1/60 20190207 15:42:00 all (16/614): umount3.sh Feb 7 15:42:32 t2 mountd[726]: bad exports list line '/mnt': symbolic link in export path or statfs failed g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) Feb 7 15:43:02 t2 syslogd: last message repeated 1 times g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) Feb 7 15:43:58 t2 syslogd: last message repeated 1 times g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5c5c43babd4367b0, error=17) mount leak: 214/274 20190207 15:47:03 all (17/614): wire_no_page.sh 20190207 15:52:07 all (18/614): unload.sh 20190207 15:52:11 all (19/614): rw.sh Feb 7 15:46:43 t2 syslogd: last message repeated 1 times mount leak: 1/275 20190207 16:02:47 all (20/614): mmap6.sh 20190207 16:07:51 all (21/614): sendfile13.sh 20190207 16:07:55 all (22/614): rename13.sh mount leak: 1/276 20190207 16:07:59 all (23/614): spin.sh mount leak: 1/277 20190207 16:08:04 all (24/614): burnin.sh mount leak: 1/278 20190207 16:28:19 all (25/614): fifo4.sh freework leak: 37/54. 20190207 16:30:24 all (26/614): holdcnt02.sh 20190207 16:30:27 all (27/614): mmap27.sh 20190207 16:32:32 all (28/614): mkfifo2c.sh mount leak: 1/279 20190207 17:03:52 all (29/614): sysctl3.sh Expensive timeout(9) function: 0xf00fa0(0x18a09c78) 0.241590181 s 20190207 17:14:47 all (30/614): shm_super.sh 20190207 17:14:50 all (31/614): snap11.sh lock order reversal: 1st 0x1dd19c9c bufwait (bufwait) @ vm/vm_pager.c:379 2nd 0x32ef631c snaplk (snaplk) @ dev/md/md.c:988 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6ddb at lockmgr_xlock_hard+0x5b #3 0x10c7632 at __lockmgr_args+0x3d2 #4 0x14364f4 at ffs_lock+0x94 #5 0x1734b69 at VOP_LOCK1_APV+0x69 #6 0x11e2871 at _vn_lock+0x61 #7 0xc5190b at mdstart_vnode+0x53b #8 0xc4f7e0 at md_kthread+0x260 #9 0x10b117e at fork_exit+0x7e #10 0xffc053ba at exit_tag+0xcd7f238a lock order reversal: 1st 0x32ef631c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:918 2nd 0x36c501a0 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1640 stack backtrace: #0 0x1161111 at witness_debugger+0x81 #1 0x1160e59 at witness_checkorder+0xa09 #2 0x10c6ddb at lockmgr_xlock_hard+0x5b #3 0x10c7632 at __lockmgr_args+0x3d2 #4 0x140f733 at ffs_snapremove+0x123 #5 0x1408895 at ffs_truncate+0x525 #6 0x143f030 at ufs_inactive+0x250 #7 0x1734801 at VOP_INACTIVE_APV+0x81 #8 0x11cee10 at vinactive+0x120 #9 0x11cf4c5 at vputx+0x355 #10 0x11dae5f at kern_unlinkat+0x28f #11 0x11dabc2 at sys_unlink+0x42 #12 0x16fe941 at syscall+0x361 #13 0xffc053a7 at exit_tag+0xcd7f2377 mount leak: 1/280 20190207 17:17:12 all (32/614): kinfo.sh 20190207 17:37:17 all (33/614): geli.sh GEOM_ELI: Device md10.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software GEOM_ELI: md10 has been killed. GEOM_ELI: Device md10.eli destroyed. mount leak: 1/281 20190207 17:42:42 all (34/614): sendfile6.sh mount leak: 1/282 20190207 17:42:46 all (35/614): fpu.sh 20190207 17:44:15 all (36/614): sendfile9.sh mount leak: 1/283 20190207 17:45:21 all (37/614): dangling.sh mount leak: 1/284 20190207 17:55:27 all (38/614): posix_fadvise3.sh 20190207 17:55:56 all (39/614): alternativeFlushPath.sh 20190207 18:16:01 all (40/614): nullfs7.sh mount leak: 1/285 20190207 18:16:07 all (41/614): sigstop2.sh 20190207 18:17:12 all (42/614): datagram.sh 20190207 18:17:17 all (43/614): mmap16.sh 20190207 18:17:20 all (44/614): tcp4.sh witness_lock_list_get: witness exhausted 20190207 18:17:58 all (45/614): jail2.sh 20190207 18:18:01 all (46/614): graid0.sh GEOM_STRIPE: Device data created (id=2514972979). GEOM_STRIPE: Disk md10 attached to data. GEOM_STRIPE: Disk md11 attached to data. GEOM_STRIPE: Disk md12 attached to data. GEOM_STRIPE: Device stripe/data activated. swap_pager: I/O error - pageout failed; blkno 12163862,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12163894,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12163926,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12163958,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12163990,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164022,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164054,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164086,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164118,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164150,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164182,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164214,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164246,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164278,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164310,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164342,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164374,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164406,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164438,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164470,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164502,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164534,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164566,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164598,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164630,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164662,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164694,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164726,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164758,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164790,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164822,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164854,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164886,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164918,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164950,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12164982,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165014,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165046,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165078,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165110,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165142,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165174,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165206,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165238,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165270,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165302,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165334,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165366,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165654,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165686,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165718,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165750,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165782,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165814,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165846,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165878,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165910,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165942,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12165974,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166006,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166038,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166070,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166102,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166134,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166166,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166198,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166230,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166262,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166294,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166326,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166358,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166390,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166422,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166454,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166486,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166518,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166550,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166582,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166614,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166646,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166678,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166710,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166742,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166774,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166806,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166838,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166870,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166902,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166934,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166966,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12166998,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167030,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167062,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167094,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167126,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167158,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167190,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167222,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167254,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167286,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167318,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167350,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167382,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167414,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167446,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167478,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167510,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167542,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167574,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167606,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167638,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167670,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167702,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167734,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167766,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167798,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167830,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167862,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167894,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167926,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167958,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12167990,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168022,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168054,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168086,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168118,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168150,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168182,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168214,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168246,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168278,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168310,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168342,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168374,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168406,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168438,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168470,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168502,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168534,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168566,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168598,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168630,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168662,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168694,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168726,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168758,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168790,size 131072, error 12 swap_pager: I/O error - pageout failed; blkno 12168822,size 131072, error 12 GEOM_STRIPE: Disk md12 removed from data. GEOM_STRIPE: Device stripe/data deactivated. GEOM_STRIPE: Disk md11 removed from data. GEOM_STRIPE: Disk md10 removed from data. GEOM_STRIPE: Device data destroyed. mount leak: 1/286 20190207 18:29:00 all (47/614): trim.sh mount leak: 3/289 20190207 19:00:22 all (48/614): unlink.sh 20190207 19:00:29 all (49/614): nullfs10.sh mount leak: 1/290 20190207 19:00:33 all (50/614): pfl4.sh 20190207 19:00:37 all (51/614): linger2.sh Feb 7 19:00:44 t2 kernel: pid 91756 (linger2), uid 1004 inumber 891 on /mnt: filesystem full Feb 7 19:00:44 t2 kernel: pid 91754 (linger2), uid 1004 inumber 888 on /mnt: filesystem full mount leak: 1/291 20190207 19:00:49 all (52/614): trim5.sh mount leak: 1/292 20190207 19:00:53 all (53/614): suj.sh mount leak: 1/293 20190207 19:21:01 all (54/614): zfs3.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (13 allocations, 22592 bytes leaked). 20190207 19:32:34 all (55/614): namecache.sh 20190207 19:47:09 all (56/614): signal0.sh 20190207 19:56:07 all (57/614): nfs15lockd3.sh 20190207 19:59:19 all (58/614): nfs.sh 20190207 20:10:11 all (59/614): radix.sh 20190207 20:10:32 all (60/614): suj9.sh mount leak: 4/297 20190207 20:35:35 all (61/614): callout_reset_on.sh 20190207 20:37:35 all (62/614): ext2fs2.sh mount leak: 1/298 20190207 20:47:43 all (63/614): gnop2.sh GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. mount leak: 4/302 20190207 20:52:57 all (64/614): ptrace5.sh 20190207 20:58:02 all (65/614): fullpath2.sh mount leak: 151/453 20190207 21:03:29 all (66/614): cpuset.sh 20190207 21:03:31 all (67/614): sendfile2.sh 20190207 21:03:59 all (68/614): ruby.sh 20190207 21:04:02 all (69/614): msdos2.sh mount leak: 1/454 20190207 21:04:06 all (70/614): tmpfs11.sh 20190207 21:20:47 all (71/614): kevent.sh 20190207 21:22:50 all (72/614): rename4.sh 20190207 21:22:54 all (73/614): procfs2.sh 20190207 21:23:07 all (74/614): racct.sh 20190207 21:25:16 all (75/614): tmpfs18.sh 20190207 21:25:19 all (76/614): mmap28.sh Feb 7 21:25:22 t2 kernel: Failed to fully fault in a core file segment at VA 0x5ca00000 with size 0x179e000 to be written at offset 0x25b000 for process mmap28 20190207 21:25:24 all (77/614): sendmsg2.sh 20190207 21:25:32 all (78/614): md7.sh warning: total configured swap (16777216 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. mount leak: 1/455 20190207 21:36:17 all (79/614): mmap22.sh 20190207 21:44:57 all (80/614): inversion.sh 20190207 21:55:01 all (81/614): contigmalloc2.sh 20190207 21:55:08 all (82/614): mmap13.sh 20190207 21:55:30 all (83/614): newfs2.sh mount leak: 28/483 20190207 22:11:22 all (84/614): tmpfs8.sh mount leak: 1/484 20190207 22:11:55 all (85/614): nullfs12.sh mount leak: 1/485 20190207 22:12:00 all (86/614): crossmp2.sh 20190207 22:34:35 all (87/614): rdgsbase.sh 20190207 22:34:37 all (88/614): write.sh Feb 7 22:49:17 t2 kernel: pid 42388 (write), uid 0 inumber 14770116 on /: filesystem full Feb 7 22:49:26 t2 kernel: pid 81117 (write), uid 0 inumber 14770142 on /: filesystem full Feb 7 22:49:26 t2 kernel: pid 81113 (write), uid 0 inumber 14770148 on /: filesystem full 20190207 22:50:01 all (89/614): pmc5.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 IAP/4/48/0x3ff IAF/3/48/0x67 20190207 22:50:06 all (90/614): oom.sh Feb 7 22:50:14 t2 kernel: pid 88415 (sort), jid 0, uid 0, was killed: out of swap space Feb 7 22:50:17 t2 kernel: pid 88413 (sort), jid 0, uid 0, was killed: out of swap space Feb 7 22:50:18 t2 kernel: pid 88416 (sort), jid 0, uid 0, was killed: out of swap space warning: total configured swap (16777216 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. 20190207 22:50:30 all (91/614): devfs2.sh 20190207 22:50:33 all (92/614): mmap12.sh 20190207 22:50:38 all (93/614): mmap31.sh 20190207 22:50:41 all (94/614): swapoff.sh warning: total configured swap (16777216 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. mount leak: 1/486 20190207 22:53:16 all (95/614): laundry.sh mount leak: 1/487 20190207 22:58:21 all (96/614): rename2.sh 20190207 23:18:24 all (97/614): mountro2.sh mount leak: 1/488 20190207 23:18:31 all (98/614): rwlock_ronly.sh 20190207 23:18:35 all (99/614): nfs15lockd2.sh Feb 7 23:18:48 t2 kernel: Failed to write core file for process nfs15lockd (error 6) 20190207 23:23:43 all (100/614): ldt2.sh 20190207 23:23:46 all (101/614): snap7.sh 20190207 23:23:50 all (102/614): crossmp3.sh 20190207 23:30:17 all (103/614): truss.sh 20190207 23:36:11 all (104/614): mknod.sh mount leak: 1/489 20190207 23:37:15 all (105/614): kevent2.sh 20190207 23:37:23 all (106/614): full.sh mount leak: 1/490 20190207 23:39:59 all (107/614): mac_chkexec.sh 20190207 23:40:01 all (108/614): unix_socket_detach2.sh 20190207 23:45:06 all (109/614): ufsbench.sh mount leak: 5/495 20190207 23:46:42 all (110/614): fts.sh mount leak: 1/496 20190208 00:08:29 all (111/614): fifo.sh mount leak: 1/497 20190208 00:08:33 all (112/614): mmap20.sh 20190208 00:08:37 all (113/614): lockd.sh 20190208 00:09:42 all (114/614): link2.sh mount leak: 101/598 20190208 00:12:43 all (115/614): mmap7.sh 20190208 00:13:31 all (116/614): fullpath.sh 20190208 00:14:36 all (117/614): fsgs.sh 20190208 00:15:40 all (118/614): nfs13.sh 20190208 00:15:44 all (119/614): pthread7.sh 20190208 00:19:25 all (120/614): suj23.sh mount leak: 1/599 20190208 00:27:03 all (121/614): mmap17.sh 20190208 00:27:07 all (122/614): pmc.sh hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 IAP/4/48/0x3ff IAF/3/48/0x67 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0xdeadc128 fault code = supervisor read data, page not present instruction pointer = 0x20:0x10d2df4 stack pointer = 0x28:0x4eabf75c frame pointer = 0x28:0x4eabf778 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 35264 (find) trap number = 12 panic: page fault cpuid = 1 time = 1549582030 KDB: stack backtrace: db_trace_self_wrapper(185bb2d,18f5f800,0,18f5f800,4eabf4f4,...) at db_trace_self_wrapper+0x2a/frame 0x4eabf4c0 kdb_backtrace(1822ef9,5c5cbece,0,4eabf57c,fffffd,...) at kdb_backtrace+0x2d/frame 0x4eabf528 vpanic(1768fb4,4eabf57c,4eabf57c,4eabf5b8,16fe166,...) at vpanic+0x141/frame 0x4eabf55c panic(1768fb4,17f6021,36b9c8c4,1,1,...) at panic+0x1b/frame 0x4eabf570 trap_fatal(48357cf8,48357ccc,4eabf640,11b53db,18a1a4a4,...) at trap_fatal+0x3a6/frame 0x4eabf5b8 trap_pfault(deadc128,304311d0,197202c0,0,1f84b,...) at trap_pfault+0x5c/frame 0x4eabf608 trap(4eabf71c,8,28,28,deadc0d8,...) at trap+0x3e6/frame 0x4eabf710 calltrap() at 0xffc0515d/frame 0x4eabf710 --- trap 0xc, eip = 0x10d2df4, esp = 0x4eabf75c, ebp = 0x4eabf778 --- _mtx_lock_indefinite_check(4eabf7b0,3c1,182c45e,3c1,0,...) at _mtx_lock_indefinite_check+0x44/frame 0x4eabf778 thread_lock_flags_(3a6f9380,0,182c45e,3c1,0,...) at thread_lock_flags_+0xe7/frame 0x4eabf7d4 sleepq_broadcast(48357d00,4,0,0,48357d6c,...) at sleepq_broadcast+0xc6/frame 0x4eabf7fc lockmgr_xunlock_hard(0,48357d6c,174831d,452,48357ccc,...) at lockmgr_xunlock_hard+0x171/frame 0x4eabf82c lockmgr_unlock_fast_path(48357d00,0,48357d6c) at lockmgr_unlock_fast_path+0x16b/frame 0x4eabf854 vop_stdunlock(4eabf8a8,48357ccc,36b9c700,48357ccc,48357d6c,...) at vop_stdunlock+0x2b/frame 0x4eabf868 VOP_UNLOCK_APV(1e0c408,4eabf8a8,17cced0,a9f,1e21c50,...) at VOP_UNLOCK_APV+0x69/frame 0x4eabf890 vputx(48357ccc,ffffffdf,312d3300,0,36b9c700,...) at vputx+0x381/frame 0x4eabf8d8 kern_statat(36b9c700,200,ffffff9c,206448b4,0,...) at kern_statat+0x12c/frame 0x4eabf9a8 sys_fstatat(36b9c700,36b9c988,1818c5a,4,4eabfaf8,...) at sys_fstatat+0x49/frame 0x4eabfab8 syscall(4eabfba8,3b,3b,3b,ffbfe648,...) at syscall+0x361/frame 0x4eabfb9c Xint0x80_syscall() at 0xffc053a7/frame 0x4eabfb9c --- syscall (552, FreeBSD ELF32, sys_fstatat), eip = 0x205de8c7, esp = 0xffbfe58c, ebp = 0xffbfe620 --- __bss_end() at 0x205de8c7 KDB: enter: panic [ thread pid 35264 tid 103659 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 35264 tid 103659 td 0x36b9c700 kdb_enter(18274d8,18274d8,0,4eabf57c,fffffd,...) at kdb_enter+0x3a/frame 0x4eabf528 vpanic(1768fb4,4eabf57c,4eabf57c,4eabf5b8,16fe166,...) at vpanic+0x15e/frame 0x4eabf55c panic(1768fb4,17f6021,36b9c8c4,1,1,...) at panic+0x1b/frame 0x4eabf570 trap_fatal(48357cf8,48357ccc,4eabf640,11b53db,18a1a4a4,...) at trap_fatal+0x3a6/frame 0x4eabf5b8 trap_pfault(deadc128,304311d0,197202c0,0,1f84b,...) at trap_pfault+0x5c/frame 0x4eabf608 trap(4eabf71c,8,28,28,deadc0d8,...) at trap+0x3e6/frame 0x4eabf710 calltrap() at 0xffc0515d/frame 0x4eabf710 --- trap 0xc, eip = 0x10d2df4, esp = 0x4eabf75c, ebp = 0x4eabf778 --- _mtx_lock_indefinite_check(4eabf7b0,3c1,182c45e,3c1,0,...) at _mtx_lock_indefinite_check+0x44/frame 0x4eabf778 thread_lock_flags_(3a6f9380,0,182c45e,3c1,0,...) at thread_lock_flags_+0xe7/frame 0x4eabf7d4 sleepq_broadcast(48357d00,4,0,0,48357d6c,...) at sleepq_broadcast+0xc6/frame 0x4eabf7fc lockmgr_xunlock_hard(0,48357d6c,174831d,452,48357ccc,...) at lockmgr_xunlock_hard+0x171/frame 0x4eabf82c lockmgr_unlock_fast_path(48357d00,0,48357d6c) at lockmgr_unlock_fast_path+0x16b/frame 0x4eabf854 vop_stdunlock(4eabf8a8,48357ccc,36b9c700,48357ccc,48357d6c,...) at vop_stdunlock+0x2b/frame 0x4eabf868 VOP_UNLOCK_APV(1e0c408,4eabf8a8,17cced0,a9f,1e21c50,...) at VOP_UNLOCK_APV+0x69/frame 0x4eabf890 vputx(48357ccc,ffffffdf,312d3300,0,36b9c700,...) at vputx+0x381/frame 0x4eabf8d8 kern_statat(36b9c700,200,ffffff9c,206448b4,0,...) at kern_statat+0x12c/frame 0x4eabf9a8 sys_fstatat(36b9c700,36b9c988,1818c5a,4,4eabfaf8,...) at sys_fstatat+0x49/frame 0x4eabfab8 syscall(4eabfba8,3b,3b,3b,ffbfe648,...) at syscall+0x361/frame 0x4eabfb9c Xint0x80_syscall() at 0xffc053a7/frame 0x4eabfb9c --- syscall (552, FreeBSD ELF32, sys_fstatat), eip = 0x205de8c7, esp = 0xffbfe58c, ebp = 0xffbfe620 --- __bss_end() at 0x205de8c7 db:1:pho1> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x8d7140 curthread = 0x8ae0700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x11bd1c00 fpcurthread = none idlethread = 0x8ae0700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc09ff0 kesp0 = 0x11bd1bf0 common_tssp = 0xffc03000 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x7052140 curthread = 0x36b9c700: pid 35264 tid 103659 "find" curpcb = 0x4eabfc00 fpcurthread = 0x36b9c700: pid 35264 "find" idlethread = 0x8ae0380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0cff0 kesp0 = 0x4eabfbf0 common_tssp = 0xffc03068 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x7057140 curthread = 0x8ae0000: pid 11 tid 100005 "idle: cpu2" curpcb = 0x11bdbc00 fpcurthread = none idlethread = 0x8ae0000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x11bdbbf0 common_tssp = 0xffc030d0 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x705c140 curthread = 0x5a667a80: pid 35269 tid 101576 "sed" curpcb = 0x5dc09c00 fpcurthread = 0x5a667a80: pid 35269 "sed" idlethread = 0x8adfa80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0fff0 kesp0 = 0x5dc09bf0 common_tssp = 0xffc03138 tlb gen = 912165359 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0x7061140 curthread = 0x8adf700: pid 11 tid 100007 "idle: cpu4" curpcb = 0x11be5c00 fpcurthread = none idlethread = 0x8adf700: tid 100007 "idle: cpu4" APIC ID = 4 currentldt = 0x50 trampstk = 0xffc10ff0 kesp0 = 0x11be5bf0 common_tssp = 0xffc031a0 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0x7066140 curthread = 0x8adf380: pid 11 tid 100008 "idle: cpu5" curpcb = 0x11beac00 fpcurthread = none idlethread = 0x8adf380: tid 100008 "idle: cpu5" APIC ID = 5 currentldt = 0x50 trampstk = 0xffc12ff0 kesp0 = 0x11beabf0 common_tssp = 0xffc03208 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0x706b140 curthread = 0x8adf000: pid 11 tid 100009 "idle: cpu6" curpcb = 0x11befc00 fpcurthread = none idlethread = 0x8adf000: tid 100009 "idle: cpu6" APIC ID = 6 currentldt = 0x50 trampstk = 0xffc13ff0 kesp0 = 0x11befbf0 common_tssp = 0xffc03270 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0x7070140 curthread = 0x8adea80: pid 11 tid 100010 "idle: cpu7" curpcb = 0x11bf4c00 fpcurthread = none idlethread = 0x8adea80: tid 100010 "idle: cpu7" APIC ID = 7 currentldt = 0x50 trampstk = 0xffc15ff0 kesp0 = 0x11bf4bf0 common_tssp = 0xffc032d8 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0x7075140 curthread = 0x3a6f9380: pid 35265 tid 101544 "find" curpcb = 0x4eb45c00 fpcurthread = 0x3a6f9380: pid 35265 "find" idlethread = 0x8ade700: tid 100011 "idle: cpu8" APIC ID = 8 currentldt = 0x50 trampstk = 0xffc16ff0 kesp0 = 0x4eb45bf0 common_tssp = 0xffc03340 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0x707a140 curthread = 0x8ade380: pid 11 tid 100012 "idle: cpu9" curpcb = 0x11bfec00 fpcurthread = none idlethread = 0x8ade380: tid 100012 "idle: cpu9" APIC ID = 9 currentldt = 0x50 trampstk = 0xffc18ff0 kesp0 = 0x11bfebf0 common_tssp = 0xffc033a8 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0x707f140 curthread = 0x8ade000: pid 11 tid 100013 "idle: cpu10" curpcb = 0xa890c00 fpcurthread = none idlethread = 0x8ade000: tid 100013 "idle: cpu10" APIC ID = 10 currentldt = 0x50 trampstk = 0xffc19ff0 kesp0 = 0xa890bf0 common_tssp = 0xffc03410 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0x7084140 curthread = 0x8af3380: pid 11 tid 100014 "idle: cpu11" curpcb = 0xa895c00 fpcurthread = none idlethread = 0x8af3380: tid 100014 "idle: cpu11" APIC ID = 11 currentldt = 0x50 trampstk = 0xffc1bff0 kesp0 = 0xa895bf0 common_tssp = 0xffc03478 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 12 dynamic pcpu = 0x7089140 curthread = 0x8af3000: pid 11 tid 100015 "idle: cpu12" curpcb = 0xa89ac00 fpcurthread = none idlethread = 0x8af3000: tid 100015 "idle: cpu12" APIC ID = 32 currentldt = 0x50 trampstk = 0xffc1cff0 kesp0 = 0xa89abf0 common_tssp = 0xffc034e0 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 13 dynamic pcpu = 0x708e140 curthread = 0x8af2a80: pid 11 tid 100016 "idle: cpu13" curpcb = 0xa89fc00 fpcurthread = none idlethread = 0x8af2a80: tid 100016 "idle: cpu13" APIC ID = 33 currentldt = 0x50 trampstk = 0xffc1eff0 kesp0 = 0xa89fbf0 common_tssp = 0xffc03548 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 14 dynamic pcpu = 0x7093140 curthread = 0x8af2700: pid 11 tid 100017 "idle: cpu14" curpcb = 0xa8a4c00 fpcurthread = none idlethread = 0x8af2700: tid 100017 "idle: cpu14" APIC ID = 34 currentldt = 0x50 trampstk = 0xffc1fff0 kesp0 = 0xa8a4bf0 common_tssp = 0xffc035b0 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 15 dynamic pcpu = 0x7098140 curthread = 0x8af2380: pid 11 tid 100018 "idle: cpu15" curpcb = 0xa8a9c00 fpcurthread = none idlethread = 0x8af2380: tid 100018 "idle: cpu15" APIC ID = 35 currentldt = 0x50 trampstk = 0xffc21ff0 kesp0 = 0xa8a9bf0 common_tssp = 0xffc03618 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 16 dynamic pcpu = 0x709d140 curthread = 0x8af2000: pid 11 tid 100019 "idle: cpu16" curpcb = 0xa8aec00 fpcurthread = none idlethread = 0x8af2000: tid 100019 "idle: cpu16" APIC ID = 36 currentldt = 0x50 trampstk = 0xffc24ff0 kesp0 = 0xa8aebf0 common_tssp = 0xffc03680 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 17 dynamic pcpu = 0x70a2140 curthread = 0x8af1a80: pid 11 tid 100020 "idle: cpu17" curpcb = 0xa8b3c00 fpcurthread = none idlethread = 0x8af1a80: tid 100020 "idle: cpu17" APIC ID = 37 currentldt = 0x50 trampstk = 0xffc25ff0 kesp0 = 0xa8b3bf0 common_tssp = 0xffc036e8 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 18 dynamic pcpu = 0x70a7140 curthread = 0x8af1700: pid 11 tid 100021 "idle: cpu18" curpcb = 0xa8b8c00 fpcurthread = none idlethread = 0x8af1700: tid 100021 "idle: cpu18" APIC ID = 38 currentldt = 0x50 trampstk = 0xffc27ff0 kesp0 = 0xa8b8bf0 common_tssp = 0xffc03750 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 19 dynamic pcpu = 0x70ac140 curthread = 0x8af1380: pid 11 tid 100022 "idle: cpu19" curpcb = 0xa8bdc00 fpcurthread = none idlethread = 0x8af1380: tid 100022 "idle: cpu19" APIC ID = 39 currentldt = 0x50 trampstk = 0xffc28ff0 kesp0 = 0xa8bdbf0 common_tssp = 0xffc037b8 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 20 dynamic pcpu = 0x70b1140 curthread = 0x8af1000: pid 11 tid 100023 "idle: cpu20" curpcb = 0xa8c2c00 fpcurthread = none idlethread = 0x8af1000: tid 100023 "idle: cpu20" APIC ID = 40 currentldt = 0x50 trampstk = 0xffc2aff0 kesp0 = 0xa8c2bf0 common_tssp = 0xffc03820 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 21 dynamic pcpu = 0x70b6140 curthread = 0x3073e000: pid 35271 tid 103069 "sed" curpcb = 0x4d7f1c00 fpcurthread = 0x3073e000: pid 35271 "sed" idlethread = 0x8ae1a80: tid 100024 "idle: cpu21" APIC ID = 41 currentldt = 0x50 trampstk = 0xffc2bff0 kesp0 = 0x4d7f1bf0 common_tssp = 0xffc03888 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 22 dynamic pcpu = 0x70bb140 curthread = 0x8ae1700: pid 11 tid 100025 "idle: cpu22" curpcb = 0xa8ccc00 fpcurthread = none idlethread = 0x8ae1700: tid 100025 "idle: cpu22" APIC ID = 42 currentldt = 0x50 trampstk = 0xffc2dff0 kesp0 = 0xa8ccbf0 common_tssp = 0xffc038f0 tlb gen = 912165360 curvnet = 0 spin locks held: cpuid = 23 dynamic pcpu = 0x70c0140 curthread = 0x8ae1380: pid 11 tid 100026 "idle: cpu23" curpcb = 0xa8d1c00 fpcurthread = none idlethread = 0x8ae1380: tid 100026 "idle: cpu23" APIC ID = 43 currentldt = 0x50 trampstk = 0xffc2eff0 kesp0 = 0xa8d1bf0 common_tssp = 0xffc03958 tlb gen = 912165360 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0x3dd6a16c: tag ufs, type VDIR usecount 5, writecount 0, refcount 8 mountedhere 0 flags (VI_ACTIVE) v_object 0x33cb4138 ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0x10c6d27 at lockmgr_slock_hard+0x2f7 #1 0x10c73a8 at __lockmgr_args+0x148 #2 0x14364f4 at ffs_lock+0x94 #3 0x1734b69 at VOP_LOCK1_APV+0x69 #4 0x11e2871 at _vn_lock+0x61 #5 0x11c1a56 at lookup+0xc6 #6 0x11c141d at namei+0x56d #7 0x11db8b5 at kern_statat+0x85 #8 0x11dc289 at sys_fstatat+0x49 #9 0x16fe941 at syscall+0x361 #10 0xffc053a7 at uncore_pcpu+0xcd7cc22b ino 13242240, on dev gpt/ssdrootfs db:1:pho1> show mount 0x19bf9000 /dev/gpt/ssdrootfs on / (ufs) 0x19bfa000 devfs on /dev (devfs) 0x31070000 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 0 Buf domain 0 freebufs 5544 lofreebufs 711 hifreebufs 1066 bufspace 5705728 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 33 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 185 (4333568) dirtyq count 33 (970752) wakeup 0 lim 4 CPU 1, 3, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 2, 1, 0, 1, 0, 3, Locked buffers: 0 space 0 Total buffers: 5779 space 5705728 db:1:pho1> show page vm_cnt.v_free_count: 576771 vm_cnt.v_inactive_count: 17712 vm_cnt.v_active_count: 2270 vm_cnt.v_laundry_count: 5 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(185bb2d,4eabedc8,4eabee02,fffffffe,0,...) at db_trace_self_wrapper+0x2a/frame 0x4eabedb0 kdb_backtrace(175bd4c,4eabef38,4eabef2c,16fd460,4eabedfe,...) at kdb_backtrace+0x2d/frame 0x4eabee18 kdb_reenter(4eabedfe,1,c,1f2c,0,...) at kdb_reenter+0x2a/frame 0x4eabee28 trap(4eabef38,8,28,28,1,...) at trap+0xe0/frame 0x4eabef2c calltrap() at 0xffc0515d/frame 0x4eabef2c --- trap 0x13, eip = 0x1151202, esp = 0x4eabef78, ebp = 0x4eabef90 --- smp_rendezvous_action(0,f3,1865e32,224,1,...) at smp_rendezvous_action+0x22/frame 0x4eabef90 smp_rendezvous_cpus(fffffd,0,1138520,0,4eabf008) at smp_rendezvous_cpus+0x121/frame 0x4eabefcc smp_rendezvous(0,1138520,0,4eabf008,0,...) at smp_rendezvous+0x32/frame 0x4eabeff0 counter_u64_fetch(899cd90,5,4eabf0e8,a16f07,114019a,...) at counter_u64_fetch+0xb1/frame 0x4eabf024 vm_page_print_page_info(114019a,0,ffffffff,4eabf060,1e53d00,...) at vm_page_print_page_info+0x67/frame 0x4eabf034 db_command(1e54bc3,1,1e545e0,1e54bc2,a1a000,...) at db_command+0x297/frame 0x4eabf0e8 db_script_exec(0,0,fffffff7,4eabf180,1e53d04,...) at db_script_exec+0x168/frame 0x4eabf154 db_command(1e54b06,0,1e54680,1e54b05,0,...) at db_command+0x297/frame 0x4eabf208 db_script_exec(4eabf2c0,0,20,4eabf2a4,1e53d04,...) at db_script_exec+0x168/frame 0x4eabf278 db_command(180a8fc,4eabf390,a19ebd,18274d8,0,...) at db_command+0x297/frame 0x4eabf32c db_command_loop(18274d8,0,4eabf357,4eabf356,180fb04,...) at db_command_loop+0x50/frame 0x4eabf338 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x4eabf390 kdb_trap(3,0,4eabf4dc,4e20f501,4eabf434,...) at kdb_trap+0x144/frame 0x4eabf3c8 trap(4eabf4dc,8,28,28,100,...) at trap+0x3cc/frame 0x4eabf4d0 calltrap() at 0xffc0515d/frame 0x4eabf4d0 --- trap 0x3, eip = 0x114019a, esp = 0x4eabf51c, ebp = 0x4eabf528 --- kdb_enter(18274d8,18274d8,0,4eabf57c,fffffd,...) at kdb_enter+0x3a/frame 0x4eabf528 vpanic(1768fb4,4eabf57c,4eabf57c,4eabf5b8,16fe166,...) at vpanic+0x15e/frame 0x4eabf55c panic(1768fb4,17f6021,36b9c8c4,1,1,...) at panic+0x1b/frame 0x4eabf570 trap_fatal(48357cf8,48357ccc,4eabf640,11b53db,18a1a4a4,...) at trap_fatal+0x3a6/frame 0x4eabf5b8 trap_pfault(deadc128,304311d0,197202c0,0,1f84b,...) at trap_pfault+0x5c/frame 0x4eabf608 trap(4eabf71c,8,28,28,deadc0d8,...) at trap+0x3e6/frame 0x4eabf710 calltrap() at 0xffc0515d/frame 0x4eabf710 --- trap 0xc, eip = 0x10d2df4, esp = 0x4eabf75c, ebp = 0x4eabf778 --- _mtx_lock_indefinite_check(4eabf7b0,3c1,182c45e,3c1,0,...) at _mtx_lock_indefinite_check+0x44/frame 0x4eabf778 thread_lock_flags_(3a6f9380,0,182c45e,3c1,0,...) at thread_lock_flags_+0xe7/frame 0x4eabf7d4 sleepq_broadcast(48357d00,4,0,0,48357d6c,...) at sleepq_broadcast+0xc6/frame 0x4eabf7fc lockmgr_xunlock_hard(0,48357d6c,174831d,452,48357ccc,...) at lockmgr_xunlock_hard+0x171/frame 0x4eabf82c lockmgr_unlock_fast_path(48357d00,0,48357d6c) at lockmgr_unlock_fast_path+0x16b/frame 0x4eabf854 vop_stdunlock(4eabf8a8,48357ccc,36b9c700,48357ccc,48357d6c,...) at vop_stdunlock+0x2b/frame 0x4eabf868 VOP_UNLOCK_APV(1e0c408,4eabf8a8,17cced0,a9f,1e21c50,...) at VOP_UNLOCK_APV+0x69/frame 0x4eabf890 vputx(48357ccc,ffffffdf,312d3300,0,36b9c700,...) at vputx+0x381/frame 0x4eabf8d8 kern_statat(36b9c700,200,ffffff9c,206448b4,0,...) at kern_statat+0x12c/frame 0x4eabf9a8 sys_fstatat(36b9c700,36b9c988,1818c5a,4,4eabfaf8,...) at sys_fstatat+0x49/frame 0x4eabfab8 syscall(4eabfba8,3b,3b,3b,ffbfe648,...) at syscall+0x361/frame 0x4eabfb9c Xint0x80_syscall() at 0xffc053a7/frame 0x4eabfb9c --- syscall (552, FreeBSD ELF32, sys_fstatat), eip = 0x205de8c7, esp = 0xffbfe58c, ebp = 0xffbfe620 --- __bss_end() at 0x205de8c7 Script command 'show page' returned error db:1:pho1> show pageq pq_free 576771 dom 0 page_cnt 742732 free 576771 pq_act 2270 pq_inact 17712 pq_laund 5 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000009 06 (000256K) | 000377 05 (000128K) | 003148 04 (000064K) | 010041 03 (000032K) | 017543 02 (000016K) | 028141 01 (000008K) | 018534 00 (000004K) | 000001 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000003 03 (000032K) | 000001 02 (000016K) | 000010 01 (000008K) | 000011 00 (000004K) | 000004 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Copyright(c) 2009 - 2012 Intel Corporation.All rights reserved. Version 2.00.1201.Copyright(c) 2010 - 2012 American Megatrends,Inc. Installed BIOS: SE5C600.86B.01.08.0003 Platform ID: W2600CR