KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r341460M: Thu Dec 6 10:05:09 CET 2018 pho@t2.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 15 boot pages configured startup_alloc from "UMA Kegs", 13 boot pages left startup_alloc from "UMA Zones", 12 boot pages left startup_alloc from "UMA Zones", 11 boot pages left startup_alloc from "UMA Zones", 10 boot pages left startup_alloc from "UMA Zones", 9 boot pages left startup_alloc from "UMA Zones", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Zones", 6 boot pages left startup_alloc from "UMA Zones", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Hash", 2 boot pages left startup_alloc from "UMA Zones", 1 boot pages left Entering uma_startup1 with 0 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.24-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66746232832 (63654 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20181031/tbfadt-850) ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard Launching APs: 12 23 11 16 1 17 10 3 2 5 14 15 18 13 8 22 7 9 21 19 20 4 6 Timecounter "TSC" frequency 1995239024 Hz quality 1000 random: entropy device external interface [ath_hal] loaded kbd1 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff81100ff0, 0) error 19 netmap: loaded module nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: numa-domain 0 on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff numa-domain 0 on acpi0 pci0: numa-domain 0 on pcib0 pcib1: irq 47 at device 1.0 numa-domain 0 on pci0 pci1: numa-domain 0 on pcib1 pcib2: irq 47 at device 2.0 numa-domain 0 on pci0 pci2: numa-domain 0 on pcib2 pcib3: irq 47 at device 3.0 numa-domain 0 on pci0 pci3: numa-domain 0 on pcib3 pcib4: irq 16 at device 17.0 numa-domain 0 on pci0 pci4: numa-domain 0 on pcib4 isci0: port 0x2000-0x20ff mem 0xd7c00000-0xd7c03fff,0xd7800000-0xd7bfffff irq 16 at device 0.0 numa-domain 0 on pci4 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xd0420000-0xd04203ff irq 22 at device 26.0 numa-domain 0 on pci0 usbus0: waiting for BIOS to give up control usbus0: timed out waiting for BIOS usbus0: EHCI version 1.0 usbus0 numa-domain 0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 hdac0: mem 0xd7f20000-0xd7f23fff irq 22 at device 27.0 numa-domain 0 on pci0 pcib5: irq 16 at device 28.0 numa-domain 0 on pci0 pci5: numa-domain 0 on pcib5 igb0: port 0x1020-0x103f mem 0xd0220000-0xd023ffff,0xd0250000-0xd0253fff irq 16 at device 0.0 numa-domain 0 on pci5 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 6 queue msgs: 9 admincnt: 1 igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: using 6 rx queues 6 tx queues igb0: Using MSIX interrupts with 7 vectors igb0: allocated for 6 tx_queues igb0: allocated for 6 rx_queues igb0: Ethernet address: 00:1e:67:56:b6:9b igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0x1000-0x101f mem 0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 17 at device 0.1 numa-domain 0 on pci5 igb1: attach_pre capping queues at 8 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 8 igb1: pxm cpus: 6 queue msgs: 9 admincnt: 1 igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: using 6 rx queues 6 tx queues igb1: Using MSIX interrupts with 7 vectors igb1: allocated for 6 tx_queues igb1: allocated for 6 rx_queues igb1: Ethernet address: 00:1e:67:56:b6:9c igb1: netmap queues/slots: TX 6/1024, RX 6/1024 pcib6: irq 17 at device 28.5 numa-domain 0 on pci0 pci6: numa-domain 0 on pcib6 pcib7: at device 0.0 numa-domain 0 on pci6 pci7: numa-domain 0 on pcib7 pci7: at device 0.0 (no driver attached) pcib8: irq 18 at device 28.6 numa-domain 0 on pci0 pci8: numa-domain 0 on pcib8 xhci0: mem 0xd0010000-0xd001ffff,0xd0000000-0xd0001fff irq 18 at device 0.0 numa-domain 0 on pci8 xhci0: 64 bytes context size, 64-bit DMA usbus1 numa-domain 0 on xhci0 usbus1: 5.0Gbps Super Speed USB v3.0 pcib9: irq 19 at device 28.7 numa-domain 0 on pci0 pci9: numa-domain 0 on pcib9 ehci1: mem 0xd0410000-0xd04103ff irq 20 at device 29.0 numa-domain 0 on pci0 usbus2: waiting for BIOS to give up control usbus2: timed out waiting for BIOS usbus2: EHCI version 1.0 usbus2 numa-domain 0 on ehci1 usbus2: 480Mbps High Speed USB v2.0 pcib10: at device 30.0 numa-domain 0 on pci0 pci10: numa-domain 0 on pcib10 isab0: at device 31.0 numa-domain 0 on pci0 isa0: numa-domain 0 on isab0 ahci0: port 0x3070-0x3077,0x3060-0x3063,0x3050-0x3057,0x3040-0x3043,0x3020-0x303f mem 0xd0400000-0xd04007ff irq 21 at device 31.2 numa-domain 0 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pcib11: numa-domain 1 on acpi0 pci11: numa-domain 1 on pcib11 pcib12: at device 0.0 numa-domain 1 on pci11 pci12: numa-domain 1 on pcib12 pcib13: irq 71 at device 1.0 numa-domain 1 on pci11 pci13: numa-domain 1 on pcib13 pcib14: irq 71 at device 2.0 numa-domain 1 on pci11 pci14: numa-domain 1 on pcib14 vgapci0: port 0xa000-0xa0ff mem 0xe0000000-0xefffffff,0xd8020000-0xd803ffff irq 56 at device 0.0 numa-domain 1 on pci14 vgapci0: Boot video device hdac1: mem 0xd8040000-0xd8043fff irq 60 at device 0.1 numa-domain 1 on pci14 pcib15: irq 71 at device 3.0 numa-domain 1 on pci11 pci15: numa-domain 1 on pcib15 acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 pcib16: on acpi0 pci16: on pcib16 pci16: at device 14.1 (no driver attached) pci16: at device 19.1 (no driver attached) pci16: at device 19.4 (no driver attached) pci16: at device 19.5 (no driver attached) pcib17: on acpi0 pci17: on pcib17 pci17: at device 14.1 (no driver attached) pci17: at device 19.1 (no driver attached) pci17: at device 19.4 (no driver attached) pci17: at device 19.5 (no driver attached) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x10 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xcffff,0xda000-0xdafff,0xdb000-0xdbfff pnpid ORM0000 on isa0 est0: numa-domain 0 on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 20,22,21,23 and 24,26 on hdaa0 pcm1: at nid 27 and 25 on hdaa0 pcm2: at nid 30 on hdaa0 hdacc1: at cad 0 on hdac1 hdaa1: at nid 1 on hdacc1 pcm3: at nid 3 on hdaa1 ugen2.1: at usbus2 ugen0.1: at usbus0 uhub0: on usbus2 ugen1.1: <0x104c XHCI root HUB> at usbus1 uhub1: on usbus0 uhub2: <0x104c XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ses0 at ahciem0 bus 0 scbus7 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number CVCV317607P0240CGN ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 228936MB (468862128 512 byte sectors) da0 at isci0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SPC-3 SCSI device da0: Serial Number 6SL64N800000N339035N da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 572325MB (1172123568 512 byte sectors) cd0 at ahcich1 bus 0 scbus2 target 0 lun 0 cd0: Removable CD-ROM SCSI device cd0: Serial Number R93E68ED300PN5 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0p2 [rw]... Expensive timeout(9) function: 0xffffffff80a19730(0xffffffff81af31f0) 0.006666642 s uhub2: 4 ports with 4 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting ddb. WARNING: reducing swap size to maximum of 65536MB per unit Starting file system checks: /dev/da0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p2: clean, 52695118 free (1148358 frags, 6443345 blocks, 1.2% fragmentation) /dev/ada0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0p2: clean, 44225573 free (275469 frags, 5493763 blocks, 0.5% fragmentation) /dev/da0p3: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p3: clean, 25238430 free (2998 frags, 3154429 blocks, 0.0% fragmentation) Mounting local file systems:. ugen2.2: at usbus2 uhub3 numa-domain 0 on uhub0 uhub3: on usbus2 ugen0.2: at usbus0 uhub4 numa-domain 0 on uhub1 uhub4: on usbus0 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat /usr/local/lib/gcc48 /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Setting hostname: t2.osted.lan. Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy:uhub4: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered . ugen2.3: at usbus2 uhub5 numa-domain 0 on uhub3 uhub5: on usbus2 uhub5: MTT enabled lo0: link state changed to UP uhub5: 4 ports with 4 removable, self powered ugen2.4: at usbus2 ukbd0 numa-domain 0 on uhub3 ukbd0: on usbus2 kbd2 at ukbd0 Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8843 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9b inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:b69b%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect status: no carrier nd6 options=23 igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e527bb ether 00:1e:67:56:b6:9c media: Ethernet autoselect status: no carrier nd6 options=29 Configuring vt: keymap. ums0 numa-domain 0 on uhub3 ums0: on usbus2 ums0: 3 buttons and [Z] coordinates ID=0 Starting ums0 moused. add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Starting casperd. lock order reversal: 1st 0xfffffe00008acc28 bufwait (bufwait) @ kern/vfs_bio.c:3916 2nd 0xfffff808f18b5400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0xffffffff80c12643 at witness_debugger+0x73 #1 0xffffffff80c12393 at witness_checkorder+0xab3 #2 0xffffffff80bb31d8 at _sx_xlock+0x68 #3 0xffffffff80ec1e4d at ufsdirhash_move+0x4d #4 0xffffffff80ec5e27 at ufs_direnter+0x327 #5 0xffffffff80ece240 at ufs_mkdir+0x8f0 #6 0xffffffff8120522c at VOP_MKDIR_APV+0x9c #7 0xffffffff80c8b8ce at kern_mkdirat+0x1be #8 0xffffffff8107bf03 at amd64_syscall+0x293 #9 0xffffffff81055dcd at fast_syscall_common+0x101 Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. sysctl: unknown oid 'debug.mnowait_failure.zalloc_whitelist' at line 13 sysctl: unknown oid 'debug.fail_point.uma_zalloc_arg' at line 14 Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Thu Dec 6 10:20 FreeBSD/amd64 (t2.osted.lan) (ttyu1) login: root Password: Dec 6 10:20:24 t2 login[3958]: ROOT LOGIN (root) ON ttyu1 Last login: Fri Nov 30 22:27:00 on ttyu1 FreeBSD 13.0-CURRENT (PHO) #0 r341460M: Thu Dec 6 10:05:09 CET 2018 You have new mail. root@t2:~ # bash $ cd ~pho/stress2/misc $ cat graid1_6.sh #!/bin/sh # # Copyright (c) 2017 Dell EMC Isilon # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD$ # # Variation of graid1_4.sh [ `id -u ` -ne 0 ] && echo "Must be root!" && exit 1 . ../default.cfg rm -f $diskimage* need=1024 # MB [ `df -k $(dirname $diskimage) | tail -1 | awk '{print int($4 / 1024)'}` -lt \ $need ] && printf "Need %d MB on %s.\n" $need `dirname $diskimage` && exit gmirror load > /dev/null 2>&1 && unload=1 [ -c /dev/mirror/test ] && { gmirror stop test; gmirror destroy test; } old=`sysctl -n kern.geom.mirror.debug` sysctl kern.geom.mirror.debug=-1 | grep -q -- -1 || sysctl kern.geom.mirror.debug=$old > /dev/null md1=$mdstart md2=$((mdstart + 1)) s=0 for u in $md1 $md2; do disk="$diskimage.$u" dd if=/dev/zero of=$disk bs=1m count=512 2>&1 | \ egrep -v "records|transferred" [ -c /dev/md$u ] && mdconfig -d -u $u mdconfig -a -t vnode -f $disk -u $u done gmirror label -v -b split -s 2048 test /dev/md$md1 /dev/md$md2 \ > /dev/null || exit 1 [ "`sysctl -in kern.geom.mirror.launch_mirror_before_timeout`" = "0" ] && sleep 5 [ -c /dev/mirror/test ] || exit 1 newfs $newfs_flags /dev/mirror/test > /dev/null mount /dev/mirror/test $mntpoint chmod 777 $mntpoint last=`tail -1 /var/log/messages | cut -c1-15` [ -z "$last" ] && last=dummy export runRUNTIME=4m export RUNDIR=$mntpoint/stressX su $testuser -c 'cd ..; ./run.sh marcus.cfg > /dev/null' & while kill -0 $! > /dev/null 2>&1; do sleep `jot -r 1 1 5` gmirror remove test md$md2 sleep `jot -r 1 1 5` gmirror insert test md$md2 done wait i=0 while ! gmirror status test | grep -q COMPLETE; do sleep 10 if [ $((i += 1)) -gt 20 ]; then echo "FAIL to COMPLETE" graid status test s=1 break fi done while mount | grep $mntpoint | grep -q /mirror/; do umount $mntpoint || sleep 1 done checkfs /dev/mirror/test || s=2 gmirror stop -f test ||s=3 gmirror destroy test 2>/dev/null [ $unload ] && gmirror unload for u in $md2 $md1; do mdconfig -d -u $u || s=4 done rm -f $diskimage* sed "1,/$last/d" < /var/log/messages | tail -20 | grep -m 1 "check-hash" && s=5 exit $s $ sh -x ./graid1_6.sh + id -u + [ 0 -ne 0 ] + . ../default.cfg + hostname + hostname=t2.osted.lan + [ -f ./t2.osted.lan ] + [ -f ../t2.osted.lan ] + . ../t2.osted.lan + testuser=stress + candidates='t1 t2 x4' + hostname + sed 's/\..*//' + me=t2 + echo t1 t2 x4 + sed s/t2// + candidates='t1 x4' + ping -c 2 -t 1 t1 + continue + ping -c 2 -t 1 x4 + BLASTHOST=x4 + nfs_export=x4:/tmp + break + export 'RUNDIR=/tmp/stressX' + export 'RUNTIME=2m' + export 'VERBOSE=1' + export 'LOAD=20' + export 'INCARNATIONS=20' + export 'BLASTHOST=x4' + export 'runRUNTIME=3d' + export 'runINCARNATIONS=1' + export 'runLOAD=100' + export 'swapINCARNATIONS=40' + export 'swapLOAD=80' + export 'syscallKILL=1' + export 'swapKILL=1' + export 'rwLOAD=70' + export 'mkdirLOAD=80' + export 'creatLOAD=80' + export 'symlinkLOAD=20' + export 'tcpKILL=1' + export 'shmINCARNATIONS=5' + [ -z '' ] + export 'TESTPROGS= testcases/rw/rw testcases/swap/swap testcases/creat/creat testcases/mkdir/mkdir testcases/thr1/thr1 testcases/udp/udp testcases/tcp/tcp ' + export 'diskimage=/tmp/diskimage' + export 'mntpoint=/mnt' + export 'testuser=stress' + export 'mdstart=10' + export 'part=a' + export 'newfs_flags=-U' + pwd + export 'stress2origin=/home/pho/stress2/misc' + dirname /home/pho/stress2/misc + export 'stress2tools=/home/pho/stress2/tools' + CC=cc + pwd + dirname /home/pho/stress2/misc + top=/home/pho/stress2 + STRESS2BIN=/home/pho/stress2/bin + id stress + rm -f '/tmp/diskimage*' + need=1024 + dirname /tmp/diskimage + tail -1 + awk '{print int($4 / 1024)}' + df -k /tmp + [ 90653 -lt 1024 ] + gmirror load + unload=1 + [ -c /dev/mirror/test ] + sysctl -n kern.geom.mirror.debug + old=0 + sysctl 'kern.geom.mirror.debug=-1' + grep -q -- -1 + md1=10 + md2=11 + s=0 + disk=/tmp/diskimage.10 + dd 'if=/dev/zero' 'of=/tmp/diskimage.10' 'bs=1m' 'count=512' + egrep -v 'records|transferred' + [ -c /dev/md10 ] + mdconfig -a -t vnode -f /tmp/diskimage.10 -u 10 + disk=/tmp/diskimage.11 + dd 'if=/dev/zero' 'of=/tmp/diskimage.11' 'bs=1m' 'count=512' + egrep -v 'records|transferred' + [ -c /dev/md11 ] + mdconfig -a -t vnode -f /tmp/diskimage.11 -u 11 + gmirror label -v -b split -s 2048 test /dev/md10 /dev/md11 lock order reversal: 1st 0xfffffe000081f430 bufwait (bufwait) @ vm/vm_pager.c:374 2nd 0xfffff8015e002068 ufs (ufs) @ dev/md/md.c:996 stack backtrace: #0 0xffffffff80c12643 at witness_debugger+0x73 #1 0xffffffff80c12393 at witness_checkorder+0xab3 #2 0xffffffff80b7ee5b at lockmgr_xlock_hard+0x6b #3 0xffffffff80b7f6be at __lockmgr_args+0x39e #4 0xffffffff80ebc3cc at ffs_lock+0x7c #5 0xffffffff81205d10 at VOP_LOCK1_APV+0x80 #6 0xffffffff80c8e810 at _vn_lock+0x60 #7 0xffffffff806bf09e at mdstart_vnode+0x44e #8 0xffffffff806bcbfe at md_kthread+0x1fe #9 0xffffffff80b6a2f4 at fork_exit+0x84 #10 0xffffffff810564ce at fork_trampoline+0xe + sysctl -in kern.geom.mirror.launch_mirror_before_timeout + [ 1 '=' 0 ] + [ -c /dev/mirror/test ] + newfs -U /dev/mirror/test + mount /dev/mirror/test /mnt + chmod 777 /mnt + tail -1 /var/log/messages + cut -c1-15 + last='Dec 6 10:21:07' + [ -z 'Dec 6 10:21:07' ] + export 'runRUNTIME=4m' + export 'RUNDIR=/mnt/stressX' + kill -0 11840 + jot -r 1 1 5 + su stress -c 'cd ..; ./run.sh marcus.cfg > /dev/null' + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 5 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 3 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 /mnt: create/symlink failed, no inodes free creat: creat(p18242.00281). creat.c:124: No space left on device creat: creat(p18244.00297). creat.c:124: openat: openat(p18240.00036), openat.c:175creat: : No space left on device creat: creat(p18241.00207). creat.c:124: No space left on device No space left on device creat(p18247.00995). creat.c:124: No space left on device Dec 6 10:24:02 t2 kernel: pid 18242 (creat), uid 1004 inumber 16399 on /mnt: out of inodes + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 5 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 3 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 4 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 4 /mnt: create/symlink failed, no inodes free creat: creat(p18245.00035). creat.c:124: No space left on device Dec 6 10:24:53 t2 kernel: pid 18245 (creat), uid 1004 inumber 49171 on /mnt: out of inodes + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 3 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 5 + gmirror remove test md11 + jot -r 1 1 5 + sleep 5 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 3 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 2 + gmirror remove test md11 + jot -r 1 1 5 + sleep 1 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 1 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + gmirror insert test md11 + kill -0 11840 + jot -r 1 1 5 + sleep 3 + gmirror remove test md11 + jot -r 1 1 5 + sleep 2 + gmirror insert test md11 + kill -0 11840 + wait + i=0 + gmirror status test + grep -q COMPLETE + sleep 10 + [ 1 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 2 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 3 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 4 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 5 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 6 -gt 20 ] + gmirror status test + grep -q COMPLETE + sleep 10 + [ 7 -gt 20 ] + gmirror status test + grep -q COMPLETE + mount + grep /mnt + grep -q /mirror/ + umount /mnt lock order reversal: 1st 0xfffff808f18e1558 ufs (ufs) @ kern/vfs_mount.c:1347 2nd 0xfffff808f1c677d0 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1588 stack backtrace: #0 0xffffffff80c12643 at witness_debugger+0x73 #1 0xffffffff80c12393 at witness_checkorder+0xab3 #2 0xffffffff80b7e989 at lockmgr_lock_fast_path+0x169 #3 0xffffffff81205d10 at VOP_LOCK1_APV+0x80 #4 0xffffffff80c8e810 at _vn_lock+0x60 #5 0xffffffff80ebaa3c at ffs_sync+0x2cc #6 0xffffffff80c75382 at dounmount+0x4c2 #7 0xffffffff80c74e20 at sys_unmount+0x300 #8 0xffffffff8107bf03 at amd64_syscall+0x293 #9 0xffffffff81055dcd at fast_syscall_common+0x101 + mount + grep /mnt + grep -q /mirror/ + checkfs /dev/mirror/test + local s + fsck -t ufs -fy /dev/mirror/test + grep -v 'IS CLEAN' /tmp/fsck.log + LANG=C egrep -q '[A-Z][A-Z]' + rm -f /tmp/fsck.log + s=0 + return 0 + gmirror stop -f test + gmirror destroy test + [ 1 ] + gmirror unload + mdconfig -d -u 11 + mdconfig -d -u 10 + rm -f /tmp/diskimage.10 /tmp/diskimage.11 + sed '1,/Dec 6 10:21:07/d' + tail -20 + grep -m 1 check-hash Dec 6 10:25:05 t2 kernel: Superblock check-hash failed: recorded check-hash 0xbd4f7161 != computed check-hash 0x65a2ae40 + s=5 + exit 5 $