GENERIC HEAD from 2010-04-19 20:41:13 UTC, r206862, vmcore.436 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0 r206854: Mon Apr 19 21:46:40 CEST 2010 pho@x4.osted.lan:/var/tmp/head_page_lock_2/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3536142336 (3372 MB) : Trying to mount root from ufs:/dev/ad4s1a WARNING: / was not properly dismounted Entropy harvesting: interrupts ethernet point_to_point kickstart. /dev/ad4s1a: 5275 files, 374024 used, 638991 free (3711 frags, 79410 blocks, 0.4% fragmentation) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585576 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585577 (28 should be 4) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585578 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585579 (8 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585580 (16 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585581 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585582 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585583 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585584 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585585 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585586 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585587 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585588 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585589 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585590 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585591 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585592 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585593 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585594 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585595 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585596 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585597 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585598 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585599 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585600 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585601 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585602 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585603 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585604 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585605 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585606 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585607 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585608 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585609 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585610 (4 should be 0) (CORRECTED) /dev/ad4s1e: INCORRECT BLOCK COUNT I=3585611 (4 should be 0) (CORRECTED) /dev/ad4s1e: ZERO LENGTH DIR I=3585576 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF DIR I=3585577 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=2048 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585578 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585579 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585580 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585581 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585582 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585583 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585584 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585585 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585586 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585587 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585588 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585589 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585590 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585591 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585592 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585593 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585594 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585595 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585596 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585597 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585598 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585599 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585600 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585601 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585602 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585603 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585604 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585605 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585606 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585607 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585608 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585609 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585610 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: ZERO LENGTH DIR I=3585611 OWNER=pho MODE=40770 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585612 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585613 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585614 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585615 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585616 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585617 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585618 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585619 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585620 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585621 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585622 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585623 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585624 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585625 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585626 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585627 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585628 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585629 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585630 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585631 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585632 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585633 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585634 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585635 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585636 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585637 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585639 OWNER=pho MODE=120777 /dev/ad4s1e: SIZE=14 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: UNREF FILE I=3585640 OWNER=pho MODE=100660 /dev/ad4s1e: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1e: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/ad4s1e: SUMMARY INFORMATION BAD (SALVAGED) /dev/ad4s1e: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/ad4s1e: 61734 files, 1484849 used, 49292226 free (2082 frags, 6161268 blocks, 0.0% fragmentation) /dev/ad4s1f: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/ad4s1f: SUMMARY INFORMATION BAD (SALVAGED) /dev/ad4s1f: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/ad4s1f: 2209210 files, 19173117 used, 3165656 free (379360 frags, 348287 blocks, 1.7% fragmentation) /dev/ad4s1d: INCORRECT BLOCK COUNT I=4639763 (28 should be 0) (CORRECTED) /dev/ad4s1d: UNREF FILE I=4592650 OWNER=root MODE=100644 /dev/ad4s1d: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1d: UNREF FILE I=4639763 OWNER=nagios MODE=100664 /dev/ad4s1d: SIZE=0 MTIME=Apr 19 22:07 2010 (CLEARED) /dev/ad4s1d: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/ad4s1d: SUMMARY INFORMATION BAD (SALVAGED) /dev/ad4s1d: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/ad4s1d: 1190186 files, 43227380 used, 7549695 free (108407 frags, 930161 blocks, 0.2% fragmentation) Additional TCP/IP options: rfc1323 extensions=NO no-ipv4-mapped-ipv6. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. Starting Network: fwe0. Starting Network: fwip0. add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 re0: link state changed to UP savecore: reboot after panic: mutex vm page queue mutex owned at ../../../vm/vm_pageout.c:1046 Apr 19 22:21:56 x4 savecore: reboot after panic: mutex vm page queue mutex ownedl at ../../../vm/ock order revervm_pageout.c:104sal: 6 1st 0xe6d17b60 bufwait (bufwait) @ kern/vfs_bio.c:2564 2nd 0xc70ec600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper(c0cbfdc3,f61827c0,c08e9645,c08d97eb,c0cc2e34,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08d97eb,c0cc2e34,c6d31098,c6d349e0,f618281c,...) at kdb_backtrace+0x29 _witness_debugger(c0cc2e34,c70ec600,c0ce5c82,c6d349e0,c0ce5914,...) at _witness_debugger+0x25 witness_checkorder(c70ec600,9,c0ce590b,11d,0,...) at witness_checkorder+0x839 _sx_xlock(c70ec600,0,c0ce590b,11d,c7b86074,...) at _sx_xlock+0x85 ufsdirhash_acquire(e6d17b00,f6182944,30,e76f15e8,f61828ec,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c7b86074,f6182944,15e8,f61828d8,f61828dc,...) at ufsdirhash_add+0x13 ufs_direnter(c7b91984,c787dd98,f6182944,f6182bd4,0,...) at ufs_direnter+0x779 ufs_makeinode(f6182bd4,c0dcc440,f6182b30,f6182a8c,c0bfc025,...) at ufs_makeinode+0x58d ufs_create(f6182b30,c0cfb7ae,0,0,f6182ba8,...) at ufs_create+0x30 VOP_CREATE_APV(c0dcbf00,f6182b30,f6182bd4,f6182ac8,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(f6182ba8,f6182c5c,180,0,c6d80380,...) at vn_open_cred+0x205 vn_open(f6182ba8,f6182c5c,180,c77f0070,c0b035f0,...) at vn_open+0x3b kern_openat(c781d480,ffffff9c,28450000,0,602,...) at kern_openat+0x125 kern_open(c781d480,28450000,0,601,180,...) at kern_open+0x35 open(c781d480,f6182cf8,c0cf7e58,c0cdf55e,c781a2a8,...) at open+0x30 syscall(f6182d38) at syscall+0x230 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x28188243, esp = 0xbfbfe2bc, ebp = 0xbfbfede8 --- savecore: writing core to vmcore.435 Writing crash summary to /var/crash/core.txt.435. Additional ABI support: linux. Performing sanity check of nagios configuration: OK lock order reversal: 1st 0xc79f85c8 pseudofs (pseudofs) @ kern/vfs_subr.c:2091 2nd 0xc0e1b81c allproc (allproc) @ kern/kern_proc.c:283 KDB: stack backtrace: db_trace_self_wrapper(c0cbfdc3,f622a860,c08e9645,c08d97eb,c0cc2e34,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08d97eb,c0cc2e34,c6d34ab0,c6d2d0d0,f622a8bc,...) at kdb_backtrace+0x29 _witness_debugger(c0cc2e34,c0e1b81c,c0cbbb09,c6d2d0d0,c0cbb9e9,...) at _witness_debugger+0x25 witness_checkorder(c0e1b81c,1,c0cbb9e0,11b,0,...) at witness_checkorder+0x839 _sx_slock(c0e1b81c,0,c0cbb9e0,11b,c70de780,...) at _sx_slock+0x85 pfind(52f,c70de3a0,4,c0cbb07c,c77ecb40,...) at pfind+0x2f pfs_visible(0,0,c0cb1fc7,7c,c79f8570,...) at pfs_visible+0x3a pfs_lookup(f622a9bc,c0cfaffb,c79f8570,c79f8570,f622ab88,...) at pfs_lookup+0x3dd VOP_CACHEDLOOKUP_APV(c0da7200,f622a9bc,f622ab88,f622ab74,c6d80380,...) at VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(f622aa50,c0ccac1f,c0de6580,200000,f622ab5c,...) at vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c0da7200,f622aa50,f622ab88,310,f622ab74,...) at VOP_LOOKUP_APV+0xe5 lookup(f622ab5c,c0cc9482,ef,ca,c77e82a8,...) at lookup+0x67b namei(f622ab5c,c188b3e8,0,c0ce798a,3b5,...) at namei+0x57f kern_statat_vnhook(c77ecb40,0,ffffff9c,bfbfe95c,0,...) at kern_statat_vnhook+0x72 kern_statat(c77ecb40,0,ffffff9c,bfbfe95c,0,...) at kern_statat+0x3c kern_stat(c77ecb40,bfbfe95c,0,f622ac18,c0f5b340,...) at kern_stat+0x36 stat(c77ecb40,f622acf8,c0cf7e58,c0cc3671,c77e82a8,...) at stat+0x2f syscall(f622ad38) at syscall+0x230 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (188, FreeBSD ELF32, stat), eip = 0x281821e3, esp = 0xbfbfe67c, ebp = 0xbfbfee18 --- Configuring syscons: keymap blanktime. Local package initialization:lock order reversal: 1st 0xc787ec94 ufs (ufs) @ kern/vfs_mount.c:1201 2nd 0xc78cf880 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1031 KDB: stack backtrace: db_trace_self_wrapper(c0cbfdc3,f618e9dc,c08e9645,c08d97eb,c0cc2e34,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08d97eb,c0cc2e34,c6d34978,c6d34840,f618ea38,...) at kdb_backtrace+0x29 _witness_debugger(c0cc2e34,c78cf880,c0cb0f71,c6d34840,c0ce3af3,...) at _witness_debugger+0x25 witness_checkorder(c78cf880,9,c0ce3aea,407,c78cf8ec,...) at witness_checkorder+0x839 __lockmgr_args(c78cf880,80400,c78cf8ec,0,0,...) at __lockmgr_args+0x804 vop_stdlock(f618eb54,4,c0cbb07c,80400,c78cf828,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c0da6640,f618eb54,c0f93e04,c0de6940,c78cf828,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c78cf828,80400,c0ce3aea,407,c79ea5a8,...) at _vn_lock+0x78 softdep_flushworklist(c79ea5a8,f618ec00,c781d240,551,c0cca9a6,...) at softdep_flushworklist+0x47 ffs_sync(c79ea5a8,1,c0cc96bb,4f6,80,...) at ffs_sync+0x2fd dounmount(c79ea5a8,8000000,c781d240,47b,ff1ca3a5,...) at dounmount+0x44e unmount(c781d240,f618ecf8,c,c781d240,c781a000,...) at unmount+0x2ff syscall(f618ed38) at syscall+0x230 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (22, FreeBSD ELF32, unmount), eip = 0x280db39f, esp = 0xbfbfe68c, ebp = 0xbfbfe758 --- ** /dev/ad4s1e ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=3585641 OWNER=pho MODE=100660 SIZE=0 MTIME=Apr 19 22:07 2010 RECONNECT? yes ** Phase 5 - Check Cyl groups 61734 files, 1484849 used, 49292226 free (2082 frags, 6161268 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** usage: kill [-s signal_name] pid ... kill -l [exit_status] kill -signal_name pid ... kill -signal_number pid ... fsck -y /tmp watchdogd. Mon Apr 19 22:23:12 CEST 2010 FreeBSD/i386 (x4.osted.lan) (console) login: Expensive timeout(9) function: 0xc077f310(0xc0fe8500) 0.012226540 s Apr 19 22:24:05 x4 su: pho to root on /dev/pts/0 Apr 19 22:49:12 x4 su: pho to root on /dev/pts/1 Stopping inetd. Shutting down local packages:. Stopping cron. Stopping sshd. Stopping apache. Waiting for PIDS: 1334. Stopping moused. Waiting for PIDS: 1278. Stopping ntpd. Stopping nagios. Stopping lockd. Stopping statd. Stopping nfsd. Stopping mountd. Stopping rpcbind. Stopping devd. Writing entropy file:. Terminated . Apr 19 22:54:49 x4 syslogd: exiting on signal 15 Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...15 11 2 3 2 0 0 done All buffers synced. lock order reversal: 1st 0xc7107c94 ufs (ufs) @ kern/vfs_mount.c:1201 2nd 0xc0e1b81c allproc (allproc) @ kern/kern_descrip.c:2518 KDB: stack backtrace: db_trace_self_wrapper(c0cbfdc3,c6b26b10,c08e9645,c08d97eb,c0cc2e34,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08d97eb,c0cc2e34,c6d34978,c6d2d0d0,c6b26b6c,...) at kdb_backtrace+0x29 _witness_debugger(c0cc2e34,c0e1b81c,c0cbbb09,c6d2d0d0,c0cb71a3,...) at _witness_debugger+0x25 witness_checkorder(c0e1b81c,1,c0cb719a,9d6,0,...) at witness_checkorder+0x839 _sx_slock(c0e1b81c,0,c0cb719a,9d6,9d,...) at _sx_slock+0x85 mountcheckdirs(c77992b8,c7107c3c,c6b26bec,4f6,c6b26bd8,...) at mountcheckdirs+0x41 dounmount(c7792000,80000,c6d7cd80,e6cea7e0,0,...) at dounmount+0x40c vfs_unmountall(c0cc9b70,0,c0cbc787,129,ac,...) at vfs_unmountall+0x4e boot(c0e1b670,0,c0cbc787,ac,bfbfe9c0,...) at boot+0x47e reboot(c6d7cd80,c6b26cf8,c0cf7e58,c0cc402a,c6d7ad48,...) at reboot+0x69 syscall(c6b26d38) at syscall+0x230 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (55, FreeBSD ELF32, reboot), eip = 0x8050973, esp = 0xbfbfe88c, ebp = 0xbfbfe968 --- Uptime: 42m53s Rebooting... cpu_reset: Stopping other CPUs (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc04d22b9 in db_fncall (dummy1=0xc08d558a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xf3f558bc "ÐXõó") at ../../../ddb/db_command.c:548 #2 0xc04d26ef in db_command (last_cmdp=0xc0de8d5c, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:445 #3 0xc04d27a4 in db_command_script (command=0xc0de9c68 "call doadump") at ../../../ddb/db_command.c:516 #4 0xc04d6970 in db_script_exec (scriptname=0xc0de95c0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc04d6a01 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc0fe8500, modif=0xf3f559f4 "") at ../../../ddb/db_script.c:375 #6 0xc04d26b1 in db_command (last_cmdp=0xc0de8d5c, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:445 #7 0xc04d280a in db_command_loop () at ../../../ddb/db_command.c:498 #8 0xc04d46ad in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc08d5406 in kdb_trap (type=0x3, code=0x0, tf=0xf3f55b9c) at ../../../kern/subr_kdb.c:535 #10 0xc0bdf14b in trap (frame=0xf3f55b9c) at ../../../i386/i386/trap.c:694 #11 0xc0bc04cb in calltrap () at ../../../i386/i386/exception.s:165 #12 0xc08d558a in kdb_enter (why=0xc0cbc8d1 "panic", msg=0xc0cbc8d1 "panic") at cpufunc.h:71 #13 0xc08a2ec6 in panic (fmt=0xc0cbb028 "mutex %s owned at %s:%d") at ../../../kern/kern_shutdown.c:562 #14 0xc0892e27 in _mtx_assert (m=0xc0f94800, what=0x0, file=0xc0ce9844 "../../../vm/vm_pageout.c", line=0x424) at ../../../kern/kern_mutex.c:719 #15 0xc0b1bb9f in vm_pageout () at ../../../vm/vm_pageout.c:1060 #16 0xc0878d98 in fork_exit (callout=0xc0b1a9a0 , arg=0x0, frame=0xf3f55d38) at ../../../kern/kern_fork.c:843 #17 0xc0bc0540 in fork_trampoline () at ../../../i386/i386/exception.s:270 (kgdb) f 14 #14 0xc0892e27 in _mtx_assert (m=0xc0f94800, what=0x0, file=0xc0ce9844 "../../../vm/vm_pageout.c", line=0x424) at ../../../kern/kern_mutex.c:719 719 panic("mutex %s owned at %s:%d", (kgdb) p *m $1 = {lock_object = {lo_name = 0xc0cc300e "vm page queue mutex", lo_flags = 0x10b0000, lo_data = 0x0, lo_witness = 0xc6d2e110}, mtx_lock = 0xc70f7480} (kgdb) up #15 0xc0b1bb9f in vm_pageout () at ../../../vm/vm_pageout.c:1060 1060 vm_page_lock_assert(m, MA_NOTOWNED); (kgdb) l 1055 vm_page_lock_queues(); 1056 } 1057 next = TAILQ_NEXT(&marker, pageq); 1058 TAILQ_REMOVE(&vm_page_queues[PQ_INACTIVE].pl, 1059 &marker, pageq); 1060 vm_page_lock_assert(m, MA_NOTOWNED); 1061 continue; 1062 } 1063 vm_page_unlock(m); 1064 VM_OBJECT_UNLOCK(object); (kgdb) info loc error = Variable "error" is not available. (kgdb) $ svn diff -x -p /var/tmp/head_page_lock_2/sys