KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r343815+42cc8428c89(sysv_sem): Wed Feb 6 08:18:52 CET 2019 pho@t2.osted.lan:/var/tmp/deviant2/sys/i386/compile/PHO i386 FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 11 boot pages configured startup_alloc from "UMA Kegs", 10 boot pages left startup_alloc from "UMA Zones", 9 boot pages left startup_alloc from "UMA Zones", 8 boot pages left startup_alloc from "UMA Hash", 7 boot pages left startup_alloc from "UMA Zones", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.24-MHz 686-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100000 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 3038281728 (2897 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20190108/tbfadt-850) ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard Launching APs: 1 22 10 15 17 21 13 12 19 4 16 6 11 18 8 5 7 14 3 23 9 20 2 Timecounter "TSC" frequency 1995237424 Hz quality 1000 random: entropy device external interface [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0x1658850, 0) error 19 kbd1 at kbdmux0 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pcib0: Ignoring 3 range above 4GB (0x380000000000-0x38007fffffff) pci0: on pcib0 pcib1: irq 47 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 47 at device 2.0 on pci0 pci2: on pcib2 pcib3: irq 47 at device 3.0 on pci0 pci3: on pcib3 pcib4: irq 16 at device 17.0 on pci0 pci4: on pcib4 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xd0420000-0xd04203ff irq 22 at device 26.0 on pci0 usbus0: waiting for BIOS to give up control usbus0: timed out waiting for BIOS usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 hdac0: mem 0xd7f20000-0xd7f23fff irq 22 at device 27.0 on pci0 pcib5: irq 16 at device 28.0 on pci0 pci5: on pcib5 igb0: port 0x1020-0x103f mem 0xd0220000-0xd023ffff,0xd0250000-0xd0253fff irq 16 at device 0.0 on pci5 igb0: PHY reset is blocked due to SOL/IDER session. igb0: Using 1024 tx descriptors and 1024 rx descriptors igb0: Using 8 rx queues 8 tx queues igb0: Using MSI-X interrupts with 9 vectors igb0: Ethernet address: 00:1e:67:56:b6:9b igb0: link state changed to UP igb1: port 0x1000-0x101f mem 0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 17 at device 0.1 on pci5 igb1: PHY reset is blocked due to SOL/IDER session. igb1: Using 1024 tx descriptors and 1024 rx descriptors igb1: Using 8 rx queues 8 tx queues igb1: Using MSI-X interrupts with 9 vectors igb1: Ethernet address: 00:1e:67:56:b6:9c pcib6: irq 17 at device 28.5 on pci0 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 pci7: at device 0.0 (no driver attached) pcib8: irq 18 at device 28.6 on pci0 pci8: on pcib8 xhci0: mem 0xd0010000-0xd001ffff,0xd0000000-0xd0001fff irq 18 at device 0.0 on pci8 xhci0: 64 bytes context size, 64-bit DMA usbus1 on xhci0 usbus1: 5.0Gbps Super Speed USB v3.0 pcib9: irq 19 at device 28.7 on pci0 pci9: on pcib9 ehci1: mem 0xd0410000-0xd04103ff irq 20 at device 29.0 on pci0 usbus2: waiting for BIOS to give up control usbus2: timed out waiting for BIOS usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 pcib10: at device 30.0 on pci0 pci10: on pcib10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x3070-0x3077,0x3060-0x3063,0x3050-0x3057,0x3040-0x3043,0x3020-0x303f mem 0xd0400000-0xd04007ff irq 21 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pcib11: on acpi0 pcib11: Ignoring 3 range above 4GB (0x380080000000-0x3800ffffffff) pci11: on pcib11 pcib12: at device 0.0 on pci11 pci12: on pcib12 pcib13: irq 71 at device 1.0 on pci11 pci13: on pcib13 pcib14: irq 71 at device 2.0 on pci11 pci14: on pcib14 vgapci0: port 0xa000-0xa0ff mem 0xe0000000-0xefffffff,0xd8020000-0xd803ffff irq 56 at device 0.0 on pci14 vgapci0: Boot video device hdac1: mem 0xd8040000-0xd8043fff irq 60 at device 0.1 on pci14 pcib15: irq 71 at device 3.0 on pci11 pci15: on pcib15 acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 pcib16: on acpi0 pci16: on pcib16 pci16: at device 14.1 (no driver attached) pci16: at device 19.1 (no driver attached) pci16: at device 19.4 (no driver attached) pci16: at device 19.5 (no driver attached) pcib17: on acpi0 pci17: on pcib17 pci17: at device 14.1 (no driver attached) pci17: at device 19.1 (no driver attached) pci17: at device 19.4 (no driver attached) pci17: at device 19.5 (no driver attached) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xcffff,0xda000-0xdafff,0xdb000-0xdbfff pnpid ORM0000 on isa0 ata0: at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata1: at port 0x170-0x177,0x376 irq 15 on isa0 ppc0: parallel port not found. est0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21,23 and 24,26 on hdaa0 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device cd0 at ahcich1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number R93E68ED300PN5 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number CVCV317607P0240CGN ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 228936MB (468862128 512 byte sectors) pcm1: at nid 27 and 25 on hdaa0 pcm2: at nid 30 on hdaa0 hdacc1: at cad 0 on hdac1 hdaa1: at nid 1 on hdacc1 pcm3: at nid 3 on hdaa1 Trying to mount root from ufs:/dev/gpt/ssdrootfs [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. WARNING: / was not properly dismounted ugen2.1: at usbus2 ugen1.1: <0x104c XHCI root HUB> at usbus1 uhub0: on usbus2 ugen0.1: at usbus0 uhub1: <0x104c XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 Expensive timeout(9) function: 0xf4bc50(0x1dbd63c) 0.007449659 s uhub2: on usbus0 Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting ddb. warning: total configured swap (16777216 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: uhub1: 4 ports with 4 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered ugen2.2: at usbus2 uhub3 on uhub0 uhub3: on usbus2 ugen0.2: at usbus0 uhub4 on uhub2 uhub4: on usbus0 uhub4: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0 on uhub4 ukbd0: on usbus0 kbd2 at ukbd0 ugen2.3: at usbus2 uhub5 on uhub3 uhub5: on usbus2 uhub5: MTT enabled uhub5: 4 ports with 4 removable, self powered ugen2.4: at usbus2 ukbd1 on uhub3 ukbd1: on usbus2 kbd3 at ukbd1 /dev/gpt/ssdrootfs: UNREF FILE I=13406155 OWNER=root MODE=100600 /dev/gpt/ssdrootfs: SIZE=0 MTIME=Feb 7 03:04 2019 (CLEARED) /dev/gpt/ssdrootfs: UNREF FILE I=13406221 OWNER=root MODE=100644 /dev/gpt/ssdrootfs: SIZE=31415910 MTIME=Feb 7 03:36 2019 (CLEARED) /dev/gpt/ssdrootfs: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/gpt/ssdrootfs: SUMMARY INFORMATION BAD (SALVAGED) /dev/gpt/ssdrootfs: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/gpt/ssdrootfs: 1159620 files, 14737531 used, 25629987 free (31107 frags, 3199860 blocks, 0.1% fragmentation) Mounting local filesystems:. lock order reversal: 1st 0x24b62a28 ufs (ufs) @ kern/vfs_subr.c:2590 2nd 0x8c12d50 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:282 3rd 0x32e905e4 ufs (ufs) @ kern/vfs_subr.c:2590 stack backtrace: #0 0x1161791 at witness_debugger+0x81 #1 0x11614d9 at witness_checkorder+0xa09 #2 0x10c723b at lockmgr_xlock_hard+0x5b #3 0x10c7a92 at __lockmgr_args+0x3d2 #4 0x1436814 at ffs_lock+0x94 #5 0x1735169 at VOP_LOCK1_APV+0x69 #6 0x11e2ef1 at _vn_lock+0x61 #7 0x11cf02e at vget+0xae #8 0x11c01dd at vfs_hash_get+0xcd #9 0x1431484 at ffs_vgetf+0x44 #10 0x14267c7 at softdep_sync_buf+0x9d7 #11 0x143764f at ffs_syncvnode+0x27f #12 0x1409051 at ffs_truncate+0x9c1 #13 0x1440f0d at ufs_direnter+0x62d #14 0x144a9c2 at ufs_makeinode+0x5a2 #15 0x1446718 at ufs_create+0x38 #16 0x1731fe5 at VOP_CREATE_APV+0x85 #17 0x11e2740 at vn_open_cred+0x260 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat /usr/local/lib/e2fsprogs /usr/local/lib/mozjpeg /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/llvm60/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: t2.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8843 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9b inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:b69b%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseT ) status: active nd6 options=23 igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Configuring vt: keymap. ums0 on uhub4 ums0: on usbus0 ums1 on uhub3 ums1: on usbus2 ums0: 18 buttons and [XYZT] coordinates ID=2 ums1: 3 buttons and [Z] coordinates ID=0 Starting ums0 moused. Starting ums1 moused. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. Creating and/or trimming log files. Starting syslogd. Starting watchdogd. No core dumps found. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0x8c3ee38 bufwait (bufwait) @ kern/vfs_bio.c:3906 2nd 0x346d3e00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x1161791 at witness_debugger+0x81 #1 0x11614d9 at witness_checkorder+0xa09 #2 0x10fec54 at _sx_xlock+0x64 #3 0x143c336 at ufsdirhash_add+0x56 #4 0x1440d63 at ufs_direnter+0x483 #5 0x1449c2f at ufs_mkdir+0x83f #6 0x1734465 at VOP_MKDIR_APV+0x85 #7 0x11df674 at kern_mkdirat+0x1d4 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. sysctl: unknown oid 'debug.mnowait_failure.zalloc_whitelist' at line 13 sysctl: unknown oid 'debug.fail_point.uma_zalloc_arg' at line 14 Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Thu Feb 7 07:43:09 CET 2019 FreeBSD/i386 (t2.osted.lan) (ttyu1) login: Feb 7 07:48:24 t2 su[949]: pho to root on /dev/pts/0 20190207 07:48:54 all (1/420): maxmemdom.sh 20190207 07:48:58 all (2/420): maxproc.sh witness_lock_list_get: witness exhausted maxproc limit exceeded by uid 65534 (pid 9297); see tuning(7) and login.conf(5) 20190207 07:49:08 all (3/420): maxvnodes.sh 20190207 07:59:40 all (4/420): md.sh Feb 7 07:59:44 t2 kernel: pid 22846 (rw), uid 0 inumber 144 on /mnt: filesystem full Feb 7 07:59:44 t2 kernel: pid 22847 (rw), uid 0 inumber 141 on /mnt: filesystem full Feb 7 07:59:44 t2 kernel: pid 22842 (rw), uid 0 inumber 143 on /mnt: filesystem full Feb 7 07:59:45 t2 kernel: pid 22838 (rw), uid 0 inumber 7 on /mnt: filesystem full Feb 7 07:59:46 t2 kernel: pid 22840 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 07:59:49 t2 kernel: pid 22837 (rw), uid 0 inumber 7 on /mnt: filesystem full Feb 7 07:59:50 t2 kernel: pid 22843 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 07:59:51 t2 kernel: pid 22849 (rw), uid 0 inumber 4 on /mnt: filesystem full Feb 7 08:01:46 t2 kernel: pid 33281 (rw), uid 0 inumber 10 on /mnt: filesystem full Feb 7 08:01:46 t2 kernel: pid 33278 (rw), uid 0 inumber 139 on /mnt: filesystem full Feb 7 08:01:49 t2 kernel: pid 33283 (rw), uid 0 inumber 139 on /mnt: filesystem full Feb 7 08:01:49 t2 kernel: pid 33280 (rw), uid 0 inumber 4 on /mnt: filesystem full Feb 7 08:01:51 t2 kernel: pid 33282 (rw), uid 0 inumber 134 on /mnt: filesystem full Feb 7 08:01:53 t2 kernel: pid 33272 (rw), uid 0 inumber 133 on /mnt: filesystem full Feb 7 08:03:49 t2 kernel: pid 42666 (rw), uid 0 inumber 10 on /mnt: filesystem full Feb 7 08:03:50 t2 kernel: pid 42660 (rw), uid 0 inumber 141 on /mnt: filesystem full Feb 7 08:05:51 t2 kernel: pid 52034 (rw), uid 0 inumber 130 on /mnt: filesystem full Feb 7 08:05:54 t2 kernel: pid 53082 (rw), uid 0 inumber 7 on /mnt: filesystem full Feb 7 08:05:57 t2 kernel: pid 53088 (rw), uid 0 inumber 137 on /mnt: filesystem full Feb 7 08:05:59 t2 kernel: pid 53084 (rw), uid 0 inumber 4 on /mnt: filesystem full Feb 7 08:06:01 t2 kernel: pid 53086 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:06:01 t2 kernel: pid 53083 (rw), uid 0 inumber 130 on /mnt: filesystem full Feb 7 08:06:05 t2 kernel: pid 54149 (rw), uid 0 inumber 140 on /mnt: filesystem full Feb 7 08:06:05 t2 kernel: pid 54154 (rw), uid 0 inumber 139 on /mnt: filesystem full Feb 7 08:06:06 t2 kernel: pid 54142 (rw), uid 0 inumber 9 on /mnt: filesystem full Feb 7 08:06:06 t2 kernel: pid 54151 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 08:06:08 t2 kernel: pid 54145 (rw), uid 0 inumber 137 on /mnt: filesystem full Feb 7 08:06:08 t2 kernel: pid 54140 (rw), uid 0 inumber 133 on /mnt: filesystem full Feb 7 08:06:10 t2 kernel: pid 54147 (rw), uid 0 inumber 4 on /mnt: filesystem full Feb 7 08:06:10 t2 kernel: pid 54150 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 08:06:12 t2 kernel: pid 54148 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 08:06:13 t2 kernel: pid 54141 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 08:07:43 t2 kernel: pid 54146 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:08:05 t2 kernel: pid 63525 (rw), uid 0 inumber 7 on /mnt: filesystem full Feb 7 08:08:09 t2 kernel: pid 63522 (rw), uid 0 inumber 135 on /mnt: filesystem full Feb 7 08:08:11 t2 kernel: pid 63518 (rw), uid 0 inumber 134 on /mnt: filesystem full Feb 7 08:08:13 t2 kernel: pid 63526 (rw), uid 0 inumber 130 on /mnt: filesystem full Feb 7 08:10:11 t2 kernel: pid 72891 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 08:10:11 t2 kernel: pid 72889 (rw), uid 0 inumber 5 on /mnt: filesystem full Feb 7 08:10:14 t2 kernel: pid 72890 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:10:16 t2 kernel: pid 73943 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 08:10:17 t2 kernel: pid 73941 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 08:10:23 t2 kernel: pid 73944 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:10:24 t2 kernel: pid 73942 (rw), uid 0 inumber 5 on /mnt: filesystem full Feb 7 08:10:26 t2 kernel: pid 74994 (rw), uid 0 inumber 135 on /mnt: filesystem full Feb 7 08:10:29 t2 kernel: pid 74993 (rw), uid 0 inumber 5 on /mnt: filesystem full Feb 7 08:10:30 t2 kernel: pid 74991 (rw), uid 0 inumber 5 on /mnt: filesystem full Feb 7 08:10:33 t2 kernel: pid 74999 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 08:10:42 t2 kernel: pid 76054 (rw), uid 0 inumber 5 on /mnt: filesystem full Feb 7 08:10:44 t2 kernel: pid 76053 (rw), uid 0 inumber 130 on /mnt: filesystem full Feb 7 08:10:47 t2 kernel: pid 76627 (rw), uid 0 inumber 141 on /mnt: filesystem full Feb 7 08:10:47 t2 kernel: pid 76637 (rw), uid 0 inumber 142 on /mnt: filesystem full Feb 7 08:10:48 t2 kernel: pid 76632 (rw), uid 0 inumber 130 on /mnt: filesystem full Feb 7 08:10:51 t2 kernel: pid 76641 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:10:52 t2 kernel: pid 76633 (rw), uid 0 inumber 6 on /mnt: filesystem full Feb 7 08:10:54 t2 kernel: pid 76629 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:10:57 t2 kernel: pid 77145 (rw), uid 0 inumber 141 on /mnt: filesystem full Feb 7 08:10:57 t2 kernel: pid 77148 (rw), uid 0 inumber 140 on /mnt: filesystem full Feb 7 08:10:59 t2 kernel: pid 77147 (rw), uid 0 inumber 140 on /mnt: filesystem full Feb 7 08:10:59 t2 kernel: pid 77146 (rw), uid 0 inumber 6 on /mnt: filesystem full Feb 7 08:10:59 t2 kernel: pid 77144 (rw), uid 0 inumber 134 on /mnt: filesystem full Feb 7 08:11:01 t2 kernel: pid 77142 (rw), uid 0 inumber 132 on /mnt: filesystem full Feb 7 08:11:03 t2 kernel: pid 77138 (rw), uid 0 inumber 131 on /mnt: filesystem full Feb 7 08:11:04 t2 kernel: pid 77149 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:11:06 t2 kernel: pid 77135 (rw), uid 0 inumber 129 on /mnt: filesystem full Feb 7 08:11:15 t2 kernel: pid 78217 (rw), uid 0 inumber 136 on /mnt: filesystem full mount leak: 1/34 20190207 08:13:08 all (5/420): md2.sh g_vfs_done():md10[WRITE(offset=118358016, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118390784, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118423552, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118456320, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118489088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118521856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118554624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118587392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118620160, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118652928, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118685696, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118718464, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=102924288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118358016, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118390784, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118423552, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118456320, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118489088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118521856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118554624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118587392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118620160, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118652928, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118685696, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118718464, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=73465856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=102924288, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118358016, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118390784, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118423552, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118456320, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118489088, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118521856, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118554624, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118587392, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118620160, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=118718464, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=73564160, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=119111680, length=32768)]error = 28 g_vfs_done():md10[WRITE(offset=70123520, length=20480)]error = 28 g_vfs_done():md10[WRITE(offset=70320128, length=12288)]error = 28 Expensive timeout(9) function: 0x132ab30(0x349a12e4) 0.007662903 s Expensive timeout(9) function: 0x132ab30(0x349a12e4) 0.021617932 s panic: ffs_read: type 0 cpuid = 15 time = 1549524203 KDB: stack backtrace: db_trace_self_wrapper(185c2d8,18f43800,0,18f43800,325cc80c,...) at db_trace_self_wrapper+0x2a/frame 0x325cc7d8 kdb_backtrace(18235ed,5c5bdceb,0,325cc894,ff7fff,...) at kdb_backtrace+0x2d/frame 0x325cc840 vpanic(18682e5,325cc894,325cc894,325cc938,1435faa,...) at vpanic+0x141/frame 0x325cc874 panic(18682e5,0,325cc8d0,11e2800,46097ccc,...) at panic+0x1b/frame 0x325cc888 ffs_read(325cc990,1785889,1e1f584,46097ccc,0,...) at ffs_read+0x68a/frame 0x325cc938 VOP_READ_APV(1e0baf8,325cc990,46097ccc,318,325cca90,...) at VOP_READ_APV+0x85/frame 0x325cc960 vn_read(31e8fe38,325cca88,191f9200,1,31ccfa80,...) at vn_read+0x221/frame 0x325cc9c0 vn_io_fault(31e8fe38,325cca88,191f9200,0,31ccfa80,...) at vn_io_fault+0x12e/frame 0x325cca0c dofileread(31e8fe38,325cca88,ffffffff,ffffffff,0,...) at dofileread+0x84/frame 0x325cca3c kern_readv(31ccfa80,4,325cca88,7f,ffbfd6b0,...) at kern_readv+0x5e/frame 0x325cca68 sys_read(31ccfa80,31ccfd08,181934e,4,0,...) at sys_read+0x54/frame 0x325ccab8 syscall(325ccba8,3b,3b,3b,1000,...) at syscall+0x361/frame 0x325ccb9c Xint0x80_syscall() at 0xffc053a7/frame 0x325ccb9c --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205f0d8f, esp = 0xffbfd5fc, ebp = 0xffbfd610 --- __bss_end() at 0x205f0d8f KDB: enter: panic [ thread pid 21537 tid 106686 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 21537 tid 106686 td 0x31ccfa80 kdb_enter(1827bcc,1827bcc,0,325cc894,ff7fff,...) at kdb_enter+0x3a/frame 0x325cc840 vpanic(18682e5,325cc894,325cc894,325cc938,1435faa,...) at vpanic+0x15e/frame 0x325cc874 panic(18682e5,0,325cc8d0,11e2800,46097ccc,...) at panic+0x1b/frame 0x325cc888 ffs_read(325cc990,1785889,1e1f584,46097ccc,0,...) at ffs_read+0x68a/frame 0x325cc938 VOP_READ_APV(1e0baf8,325cc990,46097ccc,318,325cca90,...) at VOP_READ_APV+0x85/frame 0x325cc960 vn_read(31e8fe38,325cca88,191f9200,1,31ccfa80,...) at vn_read+0x221/frame 0x325cc9c0 vn_io_fault(31e8fe38,325cca88,191f9200,0,31ccfa80,...) at vn_io_fault+0x12e/frame 0x325cca0c dofileread(31e8fe38,325cca88,ffffffff,ffffffff,0,...) at dofileread+0x84/frame 0x325cca3c kern_readv(31ccfa80,4,325cca88,7f,ffbfd6b0,...) at kern_readv+0x5e/frame 0x325cca68 sys_read(31ccfa80,31ccfd08,181934e,4,0,...) at sys_read+0x54/frame 0x325ccab8 syscall(325ccba8,3b,3b,3b,1000,...) at syscall+0x361/frame 0x325ccb9c Xint0x80_syscall() at 0xffc053a7/frame 0x325ccb9c --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205f0d8f, esp = 0xffbfd5fc, ebp = 0xffbfd610 --- __bss_end() at 0x205f0d8f db:1:pho1> show allpcpu Current CPU: 15 cpuid = 0 dynamic pcpu = 0x8d7780 curthread = 0x42092000: pid 21551 tid 104603 "creat" curpcb = 0x42295c00 fpcurthread = 0x42092000: pid 21551 "creat" idlethread = 0x8ae0700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc09ff0 kesp0 = 0x42295bf0 common_tssp = 0xffc03000 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x7052780 curthread = 0x420b9380: pid 21569 tid 104612 "tcp" curpcb = 0x422e6c00 fpcurthread = 0x420b9380: pid 21569 "tcp" idlethread = 0x8ae0380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0cff0 kesp0 = 0x422e6bf0 common_tssp = 0xffc03068 tlb gen = 32613213 curvnet = 0x8a6bf20 spin locks held: cpuid = 2 dynamic pcpu = 0x7057780 curthread = 0x420b8a80: pid 21520 tid 104614 "rw" curpcb = 0x422f8c00 fpcurthread = 0x420b8a80: pid 21520 "rw" idlethread = 0x8ae0000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x422f8bf0 common_tssp = 0xffc030d0 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x705c780 curthread = 0x3f138000: pid 21588 tid 103435 "tcp" curpcb = 0x3f50cc00 fpcurthread = 0x3f138000: pid 21588 "tcp" idlethread = 0x8adfa80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0fff0 kesp0 = 0x3f50cbf0 common_tssp = 0xffc03138 tlb gen = 32613213 curvnet = 0x8a6bf20 spin locks held: cpuid = 4 dynamic pcpu = 0x7061780 curthread = 0x430bda80: pid 21511 tid 105038 "swap" curpcb = 0x43417c00 fpcurthread = 0x430bda80: pid 21511 "swap" idlethread = 0x8adf700: tid 100007 "idle: cpu4" APIC ID = 4 currentldt = 0x50 trampstk = 0xffc10ff0 kesp0 = 0x43417bf0 common_tssp = 0xffc031a0 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0x7066780 curthread = 0x410ab000: pid 21545 tid 104220 "tcp" curpcb = 0x41279c00 fpcurthread = 0x410ab000: pid 21545 "tcp" idlethread = 0x8adf380: tid 100008 "idle: cpu5" APIC ID = 5 currentldt = 0x50 trampstk = 0xffc12ff0 kesp0 = 0x41279bf0 common_tssp = 0xffc03208 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0x706b780 curthread = 0x46090000: pid 21530 tid 106240 "rw" curpcb = 0x46282c00 fpcurthread = 0x46090000: pid 21530 "rw" idlethread = 0x8adf000: tid 100009 "idle: cpu6" APIC ID = 6 currentldt = 0x50 trampstk = 0xffc13ff0 kesp0 = 0x46282bf0 common_tssp = 0xffc03270 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0x7070780 curthread = 0x3f114000: pid 21575 tid 103412 "swap" curpcb = 0x3f44dc00 fpcurthread = 0x3f114000: pid 21575 "swap" idlethread = 0x8adea80: tid 100010 "idle: cpu7" APIC ID = 7 currentldt = 0x50 trampstk = 0xffc15ff0 kesp0 = 0x3f44dbf0 common_tssp = 0xffc032d8 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0x7075780 curthread = 0x37703380: pid 21526 tid 106702 "creat" curpcb = 0x472b9c00 fpcurthread = 0x37703380: pid 21526 "creat" idlethread = 0x8ade700: tid 100011 "idle: cpu8" APIC ID = 8 currentldt = 0x50 trampstk = 0xffc16ff0 kesp0 = 0x472b9bf0 common_tssp = 0xffc03340 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0x707a780 curthread = 0x41157000: pid 21547 tid 104286 "mkdir" curpcb = 0x414a3c00 fpcurthread = 0x41157000: pid 21547 "mkdir" idlethread = 0x8ade380: tid 100012 "idle: cpu9" APIC ID = 9 currentldt = 0x50 trampstk = 0xffc18ff0 kesp0 = 0x414a3bf0 common_tssp = 0xffc033a8 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0x707f780 curthread = 0x36bf7380: pid 21518 tid 100604 "rw" curpcb = 0x37098c00 fpcurthread = 0x36bf7380: pid 21518 "rw" idlethread = 0x8ade000: tid 100013 "idle: cpu10" APIC ID = 10 currentldt = 0x50 trampstk = 0xffc19ff0 kesp0 = 0x37098bf0 common_tssp = 0xffc03410 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0x7084780 curthread = 0x450b1a80: pid 21529 tid 105841 "mkdir" curpcb = 0x45236c00 fpcurthread = 0x450b1a80: pid 21529 "mkdir" idlethread = 0x8af3380: tid 100014 "idle: cpu11" APIC ID = 11 currentldt = 0x50 trampstk = 0xffc1bff0 kesp0 = 0x45236bf0 common_tssp = 0xffc03478 tlb gen = 32613212 curvnet = 0 spin locks held: cpuid = 12 dynamic pcpu = 0x7089780 curthread = 0x4704f380: pid 21581 tid 106661 "swap" curpcb = 0x47231c00 fpcurthread = 0x4704f380: pid 21581 "swap" idlethread = 0x8af3000: tid 100015 "idle: cpu12" APIC ID = 32 currentldt = 0x50 trampstk = 0xffc1cff0 kesp0 = 0x47231bf0 common_tssp = 0xffc034e0 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 13 dynamic pcpu = 0x708e780 curthread = 0x469eaa80: pid 21565 tid 106597 "tcp" curpcb = 0x46ecdc00 fpcurthread = 0x469eaa80: pid 21565 "tcp" idlethread = 0x8af2a80: tid 100016 "idle: cpu13" APIC ID = 33 currentldt = 0x50 trampstk = 0xffc1eff0 kesp0 = 0x46ecdbf0 common_tssp = 0xffc03548 tlb gen = 32613213 curvnet = 0x8a6bf20 spin locks held: cpuid = 14 dynamic pcpu = 0x7093780 curthread = 0x177fb380: pid 21514 tid 100254 "swap" curpcb = 0x321ccc00 fpcurthread = 0x177fb380: pid 21514 "swap" idlethread = 0x8af2700: tid 100017 "idle: cpu14" APIC ID = 34 currentldt = 0x50 trampstk = 0xffc1fff0 kesp0 = 0x321ccbf0 common_tssp = 0xffc035b0 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 15 dynamic pcpu = 0x7098780 curthread = 0x31ccfa80: pid 21537 tid 106686 "rw" curpcb = 0x325ccc00 fpcurthread = 0x31ccfa80: pid 21537 "rw" idlethread = 0x8af2380: tid 100018 "idle: cpu15" APIC ID = 35 currentldt = 0x50 trampstk = 0xffc21ff0 kesp0 = 0x325ccbf0 common_tssp = 0xffc03618 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 16 dynamic pcpu = 0x709d780 curthread = 0x4707da80: pid 21611 tid 106679 "tcp" curpcb = 0x4730bc00 fpcurthread = 0x4707da80: pid 21611 "tcp" idlethread = 0x8af2000: tid 100019 "idle: cpu16" APIC ID = 36 currentldt = 0x50 trampstk = 0xffc24ff0 kesp0 = 0x4730bbf0 common_tssp = 0xffc03680 tlb gen = 32613213 curvnet = 0x8a6bf20 spin locks held: cpuid = 17 dynamic pcpu = 0x70a2780 curthread = 0x36e7e380: pid 21544 tid 100631 "rw" curpcb = 0x3743cc00 fpcurthread = 0x36e7e380: pid 21544 "rw" idlethread = 0x8af1a80: tid 100020 "idle: cpu17" APIC ID = 37 currentldt = 0x50 trampstk = 0xffc25ff0 kesp0 = 0x3743cbf0 common_tssp = 0xffc036e8 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 18 dynamic pcpu = 0x70a7780 curthread = 0x3e9cfa80: pid 21577 tid 103288 "swap" curpcb = 0x3edf2c00 fpcurthread = 0x3e9cfa80: pid 21577 "swap" idlethread = 0x8af1700: tid 100021 "idle: cpu18" APIC ID = 38 currentldt = 0x50 trampstk = 0xffc27ff0 kesp0 = 0x3edf2bf0 common_tssp = 0xffc03750 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 19 dynamic pcpu = 0x70ac780 curthread = 0x36e10000: pid 21587 tid 100594 "tcp" curpcb = 0x3701ac00 fpcurthread = 0x36e10000: pid 21587 "tcp" idlethread = 0x8af1380: tid 100022 "idle: cpu19" APIC ID = 39 currentldt = 0x50 trampstk = 0xffc28ff0 kesp0 = 0x3701abf0 common_tssp = 0xffc037b8 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 20 dynamic pcpu = 0x70b1780 curthread = 0x408ec700: pid 21522 tid 104038 "creat" curpcb = 0x40b8dc00 fpcurthread = 0x408ec700: pid 21522 "creat" idlethread = 0x8af1000: tid 100023 "idle: cpu20" APIC ID = 40 currentldt = 0x50 trampstk = 0xffc2aff0 kesp0 = 0x40b8dbf0 common_tssp = 0xffc03820 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 21 dynamic pcpu = 0x70b6780 curthread = 0x37706000: pid 21558 tid 106704 "swap" curpcb = 0x472c3c00 fpcurthread = 0x37706000: pid 21558 "swap" idlethread = 0x8ae1a80: tid 100024 "idle: cpu21" APIC ID = 41 currentldt = 0x50 trampstk = 0xffc2bff0 kesp0 = 0x472c3bf0 common_tssp = 0xffc03888 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 22 dynamic pcpu = 0x70bb780 curthread = 0x353b4700: pid 21583 tid 100626 "swap" curpcb = 0x373ffc00 fpcurthread = 0x353b4700: pid 21583 "swap" idlethread = 0x8ae1700: tid 100025 "idle: cpu22" APIC ID = 42 currentldt = 0x50 trampstk = 0xffc2dff0 kesp0 = 0x373ffbf0 common_tssp = 0xffc038f0 tlb gen = 32613213 curvnet = 0 spin locks held: cpuid = 23 dynamic pcpu = 0x70c0780 curthread = 0x41943700: pid 21509 tid 104472 "rw" curpcb = 0x41cd2c00 fpcurthread = 0x41943700: pid 21509 "rw" idlethread = 0x8ae1380: tid 100026 "idle: cpu23" APIC ID = 43 currentldt = 0x50 trampstk = 0xffc2eff0 kesp0 = 0x41cd2bf0 common_tssp = 0xffc03958 tlb gen = 32613213 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0x4416c2d8: tag ufs, type VDIR usecount 62, writecount 0, refcount 65 mountedhere 0 flags (VI_ACTIVE) v_object 0x3844509c ref 0 pages 1 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x450b1a80 (pid 21529, mkdir, tid 105841) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11db603 at kern_rmdirat+0x83 #8 0x11df762 at sys_rmdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16640, on dev md10 vnode 0x47a36888: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x19d87750 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x408ec700 (pid 21522, creat, tid 104038) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11e25a2 at vn_open_cred+0xc2 #8 0x11e24cd at vn_open+0x3d #9 0x11d9c56 at kern_openat+0x326 #10 0x11d9f19 at sys_openat+0x49 #11 0x16fef61 at syscall+0x361 #12 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16653, on dev md10 vnode 0x48218888: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x3e9a6444 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x37703380 (pid 21526, creat, tid 106702) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11e25a2 at vn_open_cred+0xc2 #8 0x11e24cd at vn_open+0x3d #9 0x11d9c56 at kern_openat+0x326 #10 0x11d9f19 at sys_openat+0x49 #11 0x16fef61 at syscall+0x361 #12 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16656, on dev md10 vnode 0x400f7ccc: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x3638accc ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x33435000 (pid 21524, creat, tid 106706) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11e25a2 at vn_open_cred+0xc2 #8 0x11e24cd at vn_open+0x3d #9 0x11d9c56 at kern_openat+0x326 #10 0x11d9f19 at sys_openat+0x49 #11 0x16fef61 at syscall+0x361 #12 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16658, on dev md10 vnode 0x37becccc: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x38d53af8 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x41056000 (pid 21550, creat, tid 104186) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11e25a2 at vn_open_cred+0xc2 #8 0x11e24cd at vn_open+0x3d #9 0x11d9c56 at kern_openat+0x326 #10 0x11d9f19 at sys_openat+0x49 #11 0x16fef61 at syscall+0x361 #12 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16659, on dev md10 vnode 0x39dd7ccc: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x460e1ccc ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x42092000 (pid 21551, creat, tid 104603) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11e25a2 at vn_open_cred+0xc2 #8 0x11e24cd at vn_open+0x3d #9 0x11d9c56 at kern_openat+0x326 #10 0x11d9f19 at sys_openat+0x49 #11 0x16fef61 at syscall+0x361 #12 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16664, on dev md10 vnode 0x469ed9f4: tag ufs, type VREG usecount 0, writecount 0, refcount 3 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x4206d6b4 ref 0 pages 10 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x40114700 (pid 21552, rw, tid 103852) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 16751, on dev md10 vnode 0x438cc9f4: tag ufs, type VREG usecount 0, writecount 0, refcount 2 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x3f8e157c ref 0 pages 10 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x420b8a80 (pid 21520, rw, tid 104614) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 16867, on dev md10 vnode 0x3344ae38: tag ufs, type VREG usecount 0, writecount 0, refcount 2 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x34ba9c30 ref 0 pages 10 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x4707f000 (pid 21512, rw, tid 106654) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 16899, on dev md10 vnode 0x345f6888: tag ufs, type VREG usecount 0, writecount 0, refcount 3 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x37b10d68 ref 0 pages 10 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x46186a80 (pid 21541, rw, tid 106337) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 16905, on dev md10 vnode 0x19123888: tag ufs, type VREG usecount 0, writecount 0, refcount 3 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x460a71d4 ref 0 pages 10 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x460d4000 (pid 21535, rw, tid 106684) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 16999, on dev md10 vnode 0x32e71444: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags (VI_ACTIVE) v_object 0x36bdcc30 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x450b1a80 (pid 21529, mkdir, tid 105841) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db603 at kern_rmdirat+0x83 #17 0x11df762 at sys_rmdir+0x32 ino 33832, on dev md10 vnode 0x1912271c: tag ufs, type VREG usecount 0, writecount 0, refcount 3 flags (VV_NOSYNC|VI_ACTIVE|VI_DOINGINACT) v_object 0x4419f7ec ref 0 pages 10 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x46090000 (pid 21530, rw, tid 106240) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db2d3 at kern_unlinkat+0x83 #17 0x11db242 at sys_unlink+0x42 ino 17154, on dev md10 vnode 0x385aa000: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x4508aa5c ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x31ccf700 (pid 21546, mkdir, tid 106687) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11db603 at kern_rmdirat+0x83 #8 0x11df762 at sys_rmdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 17386, on dev md10 vnode 0x47ae95b0: tag ufs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x3e92b270 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x31ccf700 (pid 21546, mkdir, tid 106687) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db603 at kern_rmdirat+0x83 #17 0x11df762 at sys_rmdir+0x32 ino 17393, on dev md10 vnode 0x24af871c: tag ufs, type VREG usecount 1, writecount 1, refcount 34 flags (VI_ACTIVE) v_object 0x3af18ea0 ref 0 pages 256 cleanbuf 28 dirtybuf 4 lock type ufs: EXCL by thread 0x19cfa380 (pid 21557, rw, tid 100623) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 17753, on dev md10 vnode 0x8b2d5b0: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x384a54e0 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x41157000 (pid 21547, mkdir, tid 104286) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11db603 at kern_rmdirat+0x83 #8 0x11df762 at sys_rmdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 50309, on dev md10 vnode 0x4091d16c: tag ufs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x395bb888 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x41157000 (pid 21547, mkdir, tid 104286) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db603 at kern_rmdirat+0x83 #17 0x11df762 at sys_rmdir+0x32 ino 50276, on dev md10 vnode 0x4202be38: tag ufs, type VREG usecount 1, writecount 1, refcount 11 flags (VI_ACTIVE) v_object 0x3e8ae618 ref 0 pages 72 cleanbuf 6 dirtybuf 3 lock type ufs: EXCL by thread 0x3f0f5000 (pid 21507, rw, tid 103406) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18477, on dev md10 vnode 0x32c312d8: tag ufs, type VREG usecount 1, writecount 1, refcount 9 flags (VI_ACTIVE) v_object 0x384a5c30 ref 0 pages 49 cleanbuf 3 dirtybuf 4 lock type ufs: EXCL by thread 0x3e80c380 (pid 21510, rw, tid 103106) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18542, on dev md10 vnode 0x419ca2d8: tag ufs, type VREG usecount 1, writecount 1, refcount 7 flags (VI_ACTIVE) v_object 0x44911e04 ref 0 pages 33 cleanbuf 3 dirtybuf 2 lock type ufs: EXCL by thread 0x41943700 (pid 21509, rw, tid 104472) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18637, on dev md10 vnode 0x32215888: tag ufs, type VREG usecount 1, writecount 1, refcount 6 flags (VI_ACTIVE) v_object 0x46031a5c ref 0 pages 31 cleanbuf 0 dirtybuf 4 lock type ufs: EXCL by thread 0x323d1000 (pid 21534, rw, tid 100636) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18720, on dev md10 vnode 0x3344a16c: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x353beea0 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x3e848000 (pid 21528, mkdir, tid 103131) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11db603 at kern_rmdirat+0x83 #8 0x11df762 at sys_rmdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18751, on dev md10 vnode 0x48201e38: tag ufs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x441ad9c0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x3e848000 (pid 21528, mkdir, tid 103131) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db603 at kern_rmdirat+0x83 #17 0x11df762 at sys_rmdir+0x32 ino 18763, on dev md10 vnode 0x8e7b000: tag ufs, type VREG usecount 1, writecount 1, refcount 4 flags (VI_ACTIVE) v_object 0x4191eea0 ref 0 pages 16 cleanbuf 0 dirtybuf 2 lock type ufs: EXCL by thread 0x36bf7380 (pid 21518, rw, tid 100604) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16955, on dev md10 vnode 0x439e9e38: tag ufs, type VREG usecount 1, writecount 1, refcount 4 flags (VI_ACTIVE) v_object 0x37bef4e0 ref 0 pages 16 cleanbuf 0 dirtybuf 2 lock type ufs: EXCL by thread 0x3f113a80 (pid 21563, rw, tid 103413) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18820, on dev md10 vnode 0x8a6a16c: tag ufs, type VREG usecount 1, writecount 1, refcount 5 flags (VI_ACTIVE) v_object 0x36e044e0 ref 0 pages 17 cleanbuf 0 dirtybuf 3 lock type ufs: EXCL by thread 0x33446380 (pid 21564, rw, tid 100571) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18863, on dev md10 vnode 0x33e92000: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x3d79a4e0 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x420b7700 (pid 21555, mkdir, tid 104619) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11db603 at kern_rmdirat+0x83 #8 0x11df762 at sys_rmdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18933, on dev md10 vnode 0x31ff071c: tag ufs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0x39dc4888 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0x420b7700 (pid 21555, mkdir, tid 104619) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11cf02e at vget+0xae #6 0x11c01dd at vfs_hash_get+0xcd #7 0x1431484 at ffs_vgetf+0x44 #8 0x1431437 at ffs_vget+0x37 #9 0x144013b at ufs_lookup_ino+0xaeb #10 0x143f64a at ufs_lookup+0x2a #11 0x1731df5 at VOP_CACHEDLOOKUP_APV+0x85 #12 0x11b7aa5 at vfs_cache_lookup+0xe5 #13 0x1731c05 at VOP_LOOKUP_APV+0x85 #14 0x11c25b6 at lookup+0x5a6 #15 0x11c1a9d at namei+0x56d #16 0x11db603 at kern_rmdirat+0x83 #17 0x11df762 at sys_rmdir+0x32 ino 18948, on dev md10 vnode 0x47a89ccc: tag ufs, type VREG usecount 1, writecount 1, refcount 3 flags (VI_ACTIVE) v_object 0x449ef618 ref 0 pages 6 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x3e9f1700 (pid 21542, rw, tid 103300) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18838, on dev md10 vnode 0x24bc3e38: tag ufs, type VREG usecount 1, writecount 1, refcount 4 flags (VI_ACTIVE) v_object 0x3f1d0ea0 ref 0 pages 9 cleanbuf 0 dirtybuf 2 lock type ufs: EXCL by thread 0x36e7e380 (pid 21544, rw, tid 100631) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18950, on dev md10 vnode 0x483b3b60: tag ufs, type VREG usecount 1, writecount 1, refcount 3 flags (VI_ACTIVE) v_object 0x40151ea0 ref 0 pages 3 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0x4704fa80 (pid 21554, rw, tid 106659) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e5dbc at vn_write+0x1bc #6 0x11e57cc at vn_io_fault_doio+0x4c #7 0x11e3622 at vn_io_fault1+0x182 #8 0x11e17d8 at vn_io_fault+0x1e8 #9 0x1165bdd at dofilewrite+0x9d #10 0x116584e at kern_writev+0x5e #11 0x11657d4 at sys_write+0x54 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18956, on dev md10 vnode 0x4384b444: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x31c91d68 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x47034700 (pid 21538, mkdir, tid 106672) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18988, on dev md10 vnode 0x468efb60: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x39d89618 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x3f114a80 (pid 21561, mkdir, tid 103436) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18992, on dev md10 vnode 0x39cccb60: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x3cc01618 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x408ada80 (pid 21560, mkdir, tid 104013) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18993, on dev md10 vnode 0x3f016888: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x384a4000 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x3f138380 (pid 21533, mkdir, tid 103434) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 18998, on dev md10 vnode 0x31e8bccc: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x394b2ea0 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x353b4380 (pid 21536, mkdir, tid 100629) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 17400, on dev md10 vnode 0x4617b71c: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0x4392b888 ref 0 pages 0 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0x47034a80 (pid 21527, mkdir, tid 106667) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11c20d6 at lookup+0xc6 #6 0x11c1a9d at namei+0x56d #7 0x11df529 at kern_mkdirat+0x89 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 19001, on dev md10 vnode 0x47abb444: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0x47034700 (pid 21538, mkdir, tid 106672) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1431598 at ffs_vgetf+0x158 #3 0x1431437 at ffs_vget+0x37 #4 0x13ffa4e at ffs_valloc+0x67e #5 0x14494bd at ufs_mkdir+0xcd #6 0x1734465 at VOP_MKDIR_APV+0x85 #7 0x11df674 at kern_mkdirat+0x1d4 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 19002, on dev md10 vnode 0x191e82d8: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0x47034a80 (pid 21527, mkdir, tid 106667) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1431598 at ffs_vgetf+0x158 #3 0x1431437 at ffs_vget+0x37 #4 0x13ffa4e at ffs_valloc+0x67e #5 0x14494bd at ufs_mkdir+0xcd #6 0x1734465 at VOP_MKDIR_APV+0x85 #7 0x11df674 at kern_mkdirat+0x1d4 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 19004, on dev md10 vnode 0x44942b60: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0x3f114a80 (pid 21561, mkdir, tid 103436) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1431598 at ffs_vgetf+0x158 #3 0x1431437 at ffs_vget+0x37 #4 0x13ffa4e at ffs_valloc+0x67e #5 0x14494bd at ufs_mkdir+0xcd #6 0x1734465 at VOP_MKDIR_APV+0x85 #7 0x11df674 at kern_mkdirat+0x1d4 #8 0x11df492 at sys_mkdir+0x32 #9 0x16fef61 at syscall+0x361 #10 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 19005, on dev md10 vnode 0x3a40de38: tag ufs, type VREG usecount 1, writecount 0, refcount 1 flags (VI_ACTIVE) lock type ufs: EXCL by thread 0x408ec700 (pid 21522, creat, tid 104038) #0 0x10c7554 at lockmgr_xlock_hard+0x374 #1 0x10c7a92 at __lockmgr_args+0x3d2 #2 0x1431598 at ffs_vgetf+0x158 #3 0x1431437 at ffs_vget+0x37 #4 0x13ffa4e at ffs_valloc+0x67e #5 0x144a4d5 at ufs_makeinode+0xb5 #6 0x1446718 at ufs_create+0x38 #7 0x1731fe5 at VOP_CREATE_APV+0x85 #8 0x11e2740 at vn_open_cred+0x260 #9 0x11e24cd at vn_open+0x3d #10 0x11d9c56 at kern_openat+0x326 #11 0x11d9f19 at sys_openat+0x49 #12 0x16fef61 at syscall+0x361 #13 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 19006, on dev md10 vnode 0x46097ccc: tag ufs, type VNON usecount 1, writecount 0, refcount 1 flags (VI_ACTIVE) lock type ufs: SHARED (count 1) #0 0x10c7187 at lockmgr_slock_hard+0x2f7 #1 0x10c7808 at __lockmgr_args+0x148 #2 0x1436814 at ffs_lock+0x94 #3 0x1735169 at VOP_LOCK1_APV+0x69 #4 0x11e2ef1 at _vn_lock+0x61 #5 0x11e59aa at vn_read+0x11a #6 0x11e171e at vn_io_fault+0x12e #7 0x1165674 at dofileread+0x84 #8 0x11652fe at kern_readv+0x5e #9 0x1165284 at sys_read+0x54 #10 0x16fef61 at syscall+0x361 #11 0xffc053a7 at ums_devclass+0xcbffd3a3 ino 16713, on dev md10 db:1:pho1> show mount 0x19bf8000 /dev/gpt/ssdrootfs on / (ufs) 0x19bf9000 devfs on /dev (devfs) 0x32e3b000 procfs on /proc (procfs) 0x4570d000 /dev/md10 on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 1383 Buf domain 0 freebufs 3790 lofreebufs 711 hifreebufs 1066 bufspace 15376384 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 58 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 1906 (13504512) dirtyq count 59 (1658880) wakeup 0 lim 4 CPU 0, 1, 1, 1, 0, 2, 2, 1, 2, 1, 0, 1, 0, 1, 2, 0, 0, 1, 3, 2, 0, 0, 2, 1, Locked buffers: 26 space 622592 Total buffers: 5779 space 15376384 db:1:pho1> show page vm_cnt.v_free_count: 25563 vm_cnt.v_inactive_count: 24838 vm_cnt.v_active_count: 527820 vm_cnt.v_laundry_count: 39368 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(185c2d8,325cc0e0,325cc11a,fffffffe,0,...) at db_trace_self_wrapper+0x2a/frame 0x325cc0c8 kdb_backtrace(175c378,325cc250,325cc244,16fda80,325cc116,...) at kdb_backtrace+0x2d/frame 0x325cc130 kdb_reenter(325cc116,1,c,1f2c,0,...) at kdb_reenter+0x2a/frame 0x325cc140 trap(325cc250,8,28,28,f,...) at trap+0xe0/frame 0x325cc244 calltrap() at 0xffc0515d/frame 0x325cc244 --- trap 0x13, eip = 0x1151882, esp = 0x325cc290, ebp = 0x325cc2a8 --- smp_rendezvous_action(0,f3,18665e8,224,f,...) at smp_rendezvous_action+0x22/frame 0x325cc2a8 smp_rendezvous_cpus(ff7fff,0,1138ba0,0,325cc320) at smp_rendezvous_cpus+0x121/frame 0x325cc2e4 smp_rendezvous(0,1138ba0,0,325cc320,0,...) at smp_rendezvous+0x32/frame 0x325cc308 counter_u64_fetch(899cd90,99c8,325cc400,a16f07,114081a,...) at counter_u64_fetch+0xb1/frame 0x325cc33c vm_page_print_page_info(114081a,0,ffffffff,325cc378,1e53700,...) at vm_page_print_page_info+0x67/frame 0x325cc34c db_command(1e545c3,1,1e53fe0,1e545c2,a1a000,...) at db_command+0x297/frame 0x325cc400 db_script_exec(0,0,fffffff7,325cc498,1e53704,...) at db_script_exec+0x168/frame 0x325cc46c db_command(1e54506,0,1e54080,1e54505,0,...) at db_command+0x297/frame 0x325cc520 db_script_exec(325cc5d8,0,20,325cc5bc,1e53704,...) at db_script_exec+0x168/frame 0x325cc590 db_command(180afd5,325cc6a8,a19ebd,1827bcc,0,...) at db_command+0x297/frame 0x325cc644 db_command_loop(1827bcc,0,325cc66f,325cc66e,18101f8,...) at db_command_loop+0x50/frame 0x325cc650 db_trap(3,0,0,0,0,...) at db_trap+0x10d/frame 0x325cc6a8 kdb_trap(3,0,325cc7f4,3220c801,325cc74c,...) at kdb_trap+0x144/frame 0x325cc6e0 trap(325cc7f4,8,28,28,100,...) at trap+0x3cc/frame 0x325cc7e8 calltrap() at 0xffc0515d/frame 0x325cc7e8 --- trap 0x3, eip = 0x114081a, esp = 0x325cc834, ebp = 0x325cc840 --- kdb_enter(1827bcc,1827bcc,0,325cc894,ff7fff,...) at kdb_enter+0x3a/frame 0x325cc840 vpanic(18682e5,325cc894,325cc894,325cc938,1435faa,...) at vpanic+0x15e/frame 0x325cc874 panic(18682e5,0,325cc8d0,11e2800,46097ccc,...) at panic+0x1b/frame 0x325cc888 ffs_read(325cc990,1785889,1e1f584,46097ccc,0,...) at ffs_read+0x68a/frame 0x325cc938 VOP_READ_APV(1e0baf8,325cc990,46097ccc,318,325cca90,...) at VOP_READ_APV+0x85/frame 0x325cc960 vn_read(31e8fe38,325cca88,191f9200,1,31ccfa80,...) at vn_read+0x221/frame 0x325cc9c0 vn_io_fault(31e8fe38,325cca88,191f9200,0,31ccfa80,...) at vn_io_fault+0x12e/frame 0x325cca0c dofileread(31e8fe38,325cca88,ffffffff,ffffffff,0,...) at dofileread+0x84/frame 0x325cca3c kern_readv(31ccfa80,4,325cca88,7f,ffbfd6b0,...) at kern_readv+0x5e/frame 0x325cca68 sys_read(31ccfa80,31ccfd08,181934e,4,0,...) at sys_read+0x54/frame 0x325ccab8 syscall(325ccba8,3b,3b,3b,1000,...) at syscall+0x361/frame 0x325ccb9c Xint0x80_syscall() at 0xffc053a7/frame 0x325ccb9c --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x205f0d8f, esp = 0xffbfd5fc, ebp = 0xffbfd610 --- __bss_end() at 0x205f0d8f Script command 'show page' returned error db:1:pho1> show pageq pq_free 25563 dom 0 page_cnt 742732 free 25563 pq_act 527820 pq_inact 24838 pq_laund 39368 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000013 07 (000512K) | 000006 06 (000256K) | 000008 05 (000128K) | 000021 04 (000064K) | 000024 03 (000032K) | 000081 02 (000016K) | 000545 01 (000008K) | 003631 00 (000004K) | 008287 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000002 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Copyright(c) 2009 - 2012 Intel Corporation.All rights reserved. Version 2.00.1201.Copyright(c) 2010 - 2012 American Megatrends,Inc. Installed BIOS: SE5C600.86B.01.08.0003 Platform ID: W2600CR