GENERIC HEAD from 2013-08-21 19:40:43 UTC, r254608M, vmcore.40 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r254608M: Wed Aug 21 22:03:15 CEST 2013 pho@t2.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (1995.24-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 32154836992 (30665 MB) : Trying to mount root from ufs:/dev/da0p2 [rw]... Setting hostuuid: 2bde2bde-f4e2-e111-aab2-001e6756b69b. Setting hostid: 0x0035ff86. Starting ddb. Entropy harvesting: interrupts ethernet kickstart. Starting file system checks: /dev/da0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p2: clean, 82335317 free (87989 frags, 10280916 blocks, 0.1% fragmentation) /dev/da0p3: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/da0p3: clean, 24370785 free (1481 frags, 3046163 blocks, 0.0% fragmentation) Mounting local file systems:. Writing entropy file:. Setting hostname: t2.osted.lan. Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 igb0: flags=8843 metric 0 mtu 1500 options=401bb ether 00:1e:67:56:b6:9b inet 192.168.1.109 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::21e:67ff:fe56:b69b%igb0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (100baseTX ) status: active igb1: flags=8c02 metric 0 mtu 1500 options=401bb ether 00:1e:67:56:b6:9c nd6 options=29 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: igb1. igb1: flags=8c02 metric 0 mtu 1500 options=401bb ether 00:1e:67:56:b6:9c nd6 options=29 media: Ethernet autoselect status: no carrier Configuring keyboard: keymap. ums0: on usbus2 ums0: 3 buttons and [Z] coordinates ID=0 Starting ums0 moused. Expensive timeout(9) function: 0xffffffff80713640(0xffffffff815996d0) 0.002755470 s add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. No core dumps found. Starting rpcbind. NFS access cache time=60 rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered lock order reversal: 1st 0xfffffe07860d8db8 bufwait (bufwait) @ kern/vfs_bio.c:3053 2nd 0xfffff80067234600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e110300 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e1103b0 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e110440 _sx_xlock() at _sx_xlock+0x75/frame 0xfffffe081e110480 ufsdirhash_add() at ufsdirhash_add+0x4c/frame 0xfffffe081e1104c0 ufs_direnter() at ufs_direnter+0x688/frame 0xfffffe081e110580 ufs_mkdir() at ufs_mkdir+0x863/frame 0xfffffe081e110780 VOP_MKDIR_APV() at VOP_MKDIR_APV+0x10e/frame 0xfffffe081e1107b0 kern_mkdirat() at kern_mkdirat+0x20e/frame 0xfffffe081e1109a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e110ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e110ab0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x800931e5a, rsp = 0x7fffffffd798, rbp = 0x7fffffffdc70 --- Expensive timeout(9) function: 0xffffffff80713640(0xffffffff815996d0) 0.010627837 s Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Configuring syscons: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Local package initialization: backup watchdogd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Wed Aug 21 22:07:01 CEST 2013 FreeBSD/amd64 (t2.osted.lan) (console) login: Aug 21 22:07:22 t2 su: pho to root on /dev/pts/0 Aug 21 22:07:36 t2 su: pho to root on /dev/pts/1 20130821 22:07:42 all (1/219): nullfs15.sh lock order reversal: 1st 0xfffff80067cb4548 ufs (ufs) @ kern/vfs_subr.c:2099 2nd 0xfffffe07860f95a8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 3rd 0xfffff80014019068 ufs (ufs) @ kern/vfs_subr.c:2099 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e27c190 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e27c240 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e27c2d0 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081e27c400 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081e27c450 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081e27c480 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081e27c4f0 vget() at vget+0x70/frame 0xfffffe081e27c540 vfs_hash_get() at vfs_hash_get+0xf5/frame 0xfffffe081e27c590 ffs_vgetf() at ffs_vgetf+0x41/frame 0xfffffe081e27c620 softdep_sync_buf() at softdep_sync_buf+0x2e4/frame 0xfffffe081e27c6d0 ffs_syncvnode() at ffs_syncvnode+0x258/frame 0xfffffe081e27c750 ffs_sync() at ffs_sync+0x212/frame 0xfffffe081e27c820 dounmount() at dounmount+0x360/frame 0xfffffe081e27c8a0 sys_unmount() at sys_unmount+0x376/frame 0xfffffe081e27c9a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e27cab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e27cab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800885dca, rsp = 0x7fffffffcf18, rbp = 0x7fffffffd030 --- lock order reversal: 1st 0xfffff800671c2068 ufs (ufs) @ kern/vfs_mount.c:1237 2nd 0xfffff80067ae7548 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1903 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e27c3c0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e27c470 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e27c500 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081e27c630 vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081e27c650 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081e27c680 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081e27c6f0 softdep_flushworklist() at softdep_flushworklist+0x70/frame 0xfffffe081e27c750 ffs_sync() at ffs_sync+0x29d/frame 0xfffffe081e27c820 dounmount() at dounmount+0x360/frame 0xfffffe081e27c8a0 sys_unmount() at sys_unmount+0x376/frame 0xfffffe081e27c9a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e27cab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e27cab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800885dca, rsp = 0x7fffffffcf18, rbp = 0x7fffffffd030 --- 20130821 22:07:43 all (2/219): snap9.sh 20130821 22:07:44 all (3/219): rename6.sh 20130821 22:07:51 all (4/219): trim.sh Expensive timeout(9) function: 0xffffffff80b2f5b0(0) 0.029218444 s ffs_fsync: dirty 0xfffff80276b3f750: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff80335a32000 ref 0 pages 0 lock type ufs: EXCL by thread 0xfffff80265f7c000 (pid 10918, creat, tid 100486) #0 0xffffffff8089f0d7 at __lockmgr_args+0x10b7 #1 0xffffffff80b122b2 at ffs_lock+0x92 #2 0xffffffff80d68fd5 at VOP_LOCK1_APV+0xf5 #3 0xffffffff80976406 at _vn_lock+0xc6 #4 0xffffffff8095b161 at lookup+0x101 #5 0xffffffff8095ad14 at namei+0x4b4 #6 0xffffffff809759e4 at vn_open_cred+0xc4 #7 0xffffffff8096f355 at kern_openat+0x1f5 #8 0xffffffff80cbe202 at amd64_syscall+0x282 #9 0xffffffff80ca14cb at Xfast_syscall+0xfb ino 32817, on dev md5a 20130821 22:44:28 all (5/219): syscall3.sh 20130821 22:54:29 all (6/219): nullfs3.sh 20130821 22:54:31 all (7/219): snap.sh lock order reversal: 1st 0xfffffe078610d8c8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:262 2nd 0xfffff805b3003d30 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2297 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e5f81f0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e5f82a0 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e5f8330 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081e5f8460 ffs_copyonwrite() at ffs_copyonwrite+0x17a/frame 0xfffffe081e5f8510 ffs_geom_strategy() at ffs_geom_strategy+0xdd/frame 0xfffffe081e5f8540 ufs_strategy() at ufs_strategy+0x8a/frame 0xfffffe081e5f8570 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0x119/frame 0xfffffe081e5f85a0 VOP_STRATEGY() at VOP_STRATEGY+0x39/frame 0xfffffe081e5f85e0 bufstrategy() at bufstrategy+0xd7/frame 0xfffffe081e5f8620 bstrategy() at bstrategy+0xf5/frame 0xfffffe081e5f8640 bufwrite() at bufwrite+0x228/frame 0xfffffe081e5f8680 ffs_syncvnode() at ffs_syncvnode+0x303/frame 0xfffffe081e5f8700 ffs_fsync() at ffs_fsync+0x20/frame 0xfffffe081e5f8730 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x10e/frame 0xfffffe081e5f8760 ufs_remove() at ufs_remove+0xdc/frame 0xfffffe081e5f87c0 VOP_REMOVE_APV() at VOP_REMOVE_APV+0x12d/frame 0xfffffe081e5f87f0 kern_unlinkat() at kern_unlinkat+0x21c/frame 0xfffffe081e5f89a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e5f8ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e5f8ab0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80093169a, rsp = 0x7fffffffd5e8, rbp = 0x7fffffffd680 --- lock order reversal: 1st 0xfffff805b3003d30 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:968 2nd 0xfffff80647a53a28 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1628 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e5f8210 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e5f82c0 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e5f8350 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081e5f8480 ffs_snapremove() at ffs_snapremove+0xfa/frame 0xfffffe081e5f84f0 ffs_truncate() at ffs_truncate+0x27e/frame 0xfffffe081e5f86d0 ufs_inactive() at ufs_inactive+0x171/frame 0xfffffe081e5f8710 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x10e/frame 0xfffffe081e5f8740 vinactive() at vinactive+0x157/frame 0xfffffe081e5f8790 vputx() at vputx+0x370/frame 0xfffffe081e5f87f0 kern_unlinkat() at kern_unlinkat+0x256/frame 0xfffffe081e5f89a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e5f8ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e5f8ab0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80093169a, rsp = 0x7fffffffd5e8, rbp = 0x7fffffffd680 --- 20130821 22:54:52 all (8/219): quota4.sh lock order reversal: 1st 0xfffff8020dd7b2d8 syncer (syncer) @ kern/vfs_subr.c:1718 2nd 0xfffff806c65e4a28 ufs (ufs) @ kern/vfs_subr.c:2099 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081de844b0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081de84560 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081de845f0 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081de84720 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081de84770 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081de847a0 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081de84810 vget() at vget+0x70/frame 0xfffffe081de84860 qsync() at qsync+0xe0/frame 0xfffffe081de848a0 ffs_sync() at ffs_sync+0x4d0/frame 0xfffffe081de84970 sync_fsync() at sync_fsync+0x129/frame 0xfffffe081de849a0 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x10e/frame 0xfffffe081de849d0 sched_sync() at sched_sync+0x34c/frame 0xfffffe081de84a70 fork_exit() at fork_exit+0x84/frame 0xfffffe081de84ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe081de84ab0 --- trap 0, rip = 0, rsp = 0xfffffe081de84b70, rbp = 0 --- lock order reversal: 1st 0xfffff80014b2fc88 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:714 2nd 0xfffff80691936790 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:720 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e5da510 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e5da5c0 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e5da650 __mtx_lock_flags() at __mtx_lock_flags+0xa6/frame 0xfffffe081e5da6a0 quotaoff_inchange() at quotaoff_inchange+0x83/frame 0xfffffe081e5da6e0 ffs_flushfiles() at ffs_flushfiles+0x7e/frame 0xfffffe081e5da750 softdep_flushfiles() at softdep_flushfiles+0x17f/frame 0xfffffe081e5da7b0 ffs_unmount() at ffs_unmount+0x1a2/frame 0xfffffe081e5da820 dounmount() at dounmount+0x39e/frame 0xfffffe081e5da8a0 sys_unmount() at sys_unmount+0x376/frame 0xfffffe081e5da9a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e5daab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e5daab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800885dca, rsp = 0x7fffffffcf18, rbp = 0x7fffffffd030 --- 20130821 22:56:12 all (9/219): md3.sh 20130821 23:07:57 all (10/219): namecache.sh 20130821 23:23:04 all (11/219): suj32.sh 20130821 23:23:06 all (12/219): tmpfs5.sh 20130821 23:23:18 all (13/219): inversion.sh 20130821 23:53:18 all (14/219): tmpfs10.sh 20130821 23:55:46 all (15/219): snap2.sh 20130821 23:55:51 all (16/219): sendfile6.sh lock order reversal: 1st 0xfffff80067351738 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:145 2nd 0xfffff8007e9d6548 ufs (ufs) @ kern/uipc_syscalls.c:2190 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e156390 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e156440 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e1564d0 __lockmgr_args() at __lockmgr_args+0x456/frame 0xfffffe081e156600 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081e156650 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081e156680 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081e1566f0 vn_sendfile() at vn_sendfile+0x738/frame 0xfffffe081e156910 do_sendfile() at do_sendfile+0x116/frame 0xfffffe081e1569a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e156ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e156ab0 --- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x80088288a, rsp = 0x7fffffffd6f8, rbp = 0x7fffffffd710 --- 20130821 23:55:53 all (17/219): ldt2.sh 20130821 23:55:53 all (18/219): nfs5.sh Aug 21 23:56:44 t2 kernel: pid 892 (nfsd), uid 0 inumber 12293 on /mnt: filesystem full 20130821 23:57:11 all (19/219): snap7.sh Aug 22 00:01:47 t2 mountd[883]: bad exports list line /mnt Aug 22 00:22:01 t2 last message repeated 3 times 20130822 00:27:13 all (20/219): callout_reset_on.sh Aug 22 00:26:01 t2 last message repeated 2 times 20130822 00:34:27 all (21/219): mac_chkexec.sh 20130822 00:34:32 all (22/219): kevent5.sh lock order reversal: 1st 0xfffff801040b5040 filedesc structure (filedesc structure) @ kern/kern_descrip.c:1184 2nd 0xfffff80553dcaa28 ufs (ufs) @ kern/vfs_subr.c:4343 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081e86d570 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081e86d620 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081e86d6b0 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081e86d7e0 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081e86d830 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081e86d860 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081e86d8d0 knlist_remove_kq() at knlist_remove_kq+0x82/frame 0xfffffe081e86d900 knote_fdclose() at knote_fdclose+0xc8/frame 0xfffffe081e86d950 closefp() at closefp+0x64/frame 0xfffffe081e86d9a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081e86dab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081e86dab0 --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x80094233a, rsp = 0x7fffffffd698, rbp = 0x7fffffffd700 --- 20130822 00:40:57 all (23/219): tmpfs7.sh 20130822 00:40:58 all (24/219): kinfo3.sh 20130822 00:56:08 all (25/219): sendfile5.sh 20130822 00:56:55 all (26/219): libMicro.sh 20130822 00:57:56 all (27/219): linger3.sh 20130822 00:58:08 all (28/219): extattr_set_fd.sh 20130822 00:58:08 all (29/219): devfs.sh 20130822 00:58:20 all (30/219): suj33.sh 20130822 00:58:46 all (31/219): mount2.sh 20130822 01:12:47 all (32/219): quota8.sh 20130822 01:18:48 all (33/219): softupdate.sh 20130822 01:40:53 all (34/219): zfs5.sh ZFS filesystem version: 5 ZFS storage pool version: features support (5000) lock order reversal: 1st 0xfffff8063efffc98 zfs (zfs) @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1805 2nd 0xfffff803c6880a28 ufs (ufs) @ kern/vfs_vnops.c:471 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081f058970 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081f058a20 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081f058ab0 __lockmgr_args() at __lockmgr_args+0x456/frame 0xfffffe081f058be0 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081f058c30 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xf5/frame 0xfffffe081f058c60 _vn_lock() at _vn_lock+0xc6/frame 0xfffffe081f058cd0 vn_rdwr() at vn_rdwr+0x1d1/frame 0xfffffe081f058d90 vdev_file_io_start() at vdev_file_io_start+0x16f/frame 0xfffffe081f058e10 zio_vdev_io_start() at zio_vdev_io_start+0x40a/frame 0xfffffe081f058e70 zio_execute() at zio_execute+0x1b6/frame 0xfffffe081f058eb0 vdev_mirror_io_start() at vdev_mirror_io_start+0x2df/frame 0xfffffe081f058f20 zio_vdev_io_start() at zio_vdev_io_start+0x1f8/frame 0xfffffe081f058f80 zio_execute() at zio_execute+0x1b6/frame 0xfffffe081f058fc0 arc_read() at arc_read+0x8df/frame 0xfffffe081f059060 dbuf_read() at dbuf_read+0x790/frame 0xfffffe081f059110 dmu_buf_hold() at dmu_buf_hold+0xc3/frame 0xfffffe081f059160 zap_lockdir() at zap_lockdir+0x4a/frame 0xfffffe081f059210 zap_lookup_norm() at zap_lookup_norm+0x3d/frame 0xfffffe081f059270 zap_lookup() at zap_lookup+0x2e/frame 0xfffffe081f0592a0 zfs_dirent_lock() at zfs_dirent_lock+0x3fd/frame 0xfffffe081f059360 zfs_dirlook() at zfs_dirlook+0x1b2/frame 0xfffffe081f0593d0 zfs_lookup() at zfs_lookup+0x2d7/frame 0xfffffe081f059460 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x91/frame 0xfffffe081f0595a0 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x108/frame 0xfffffe081f0595d0 vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe081f059620 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x108/frame 0xfffffe081f059650 lookup() at lookup+0x590/frame 0xfffffe081f0596e0 namei() at namei+0x4b4/frame 0xfffffe081f059790 kern_statat_vnhook() at kern_statat_vnhook+0x99/frame 0xfffffe081f059900 sys_lstat() at sys_lstat+0x30/frame 0xfffffe081f0599a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081f059ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081f059ab0 --- syscall (190, FreeBSD ELF64, sys_lstat), rip = 0x8019c4e1a, rsp = 0x7fffffffc4a8, rbp = 0x7fffffffce70 --- 20130822 01:52:21 all (35/219): kinfo2.sh 20130822 02:22:23 all (36/219): sem.sh 20130822 02:22:39 all (37/219): devfs2.sh 20130822 02:22:40 all (38/219): procfs.sh Expensive timeout(9) function: 0xffffffff80762f70(0xfffffe0001166cd8) 0.161015630 s Aug 22 02:27:18 t2 last message repeated 2 times 20130822 02:27:29 all (39/219): lockf2.sh 20130822 02:32:53 all (40/219): crossmp.sh Aug 22 02:37:54 t2 last message repeated 2 times 20130822 02:46:04 all (41/219): rename8.sh 20130822 02:46:32 all (42/219): sendfile.sh 20130822 02:53:17 all (43/219): ffs_blkfree.sh 20130822 02:53:46 all (44/219): rename2.sh 20130822 03:22:00 all (45/219): kevent.sh 20130822 03:22:42 all (46/219): zfs.sh 20130822 03:53:38 all (47/219): lockf3.sh 20130822 04:25:49 all (48/219): nfs6.sh Aug 22 04:26:17 t2 kernel: pid 892 (nfsd), uid 0 inumber 8857 on /mnt: filesystem full 20130822 04:27:27 all (49/219): f_offset.sh 20130822 04:28:54 all (50/219): swap2.sh 20130822 04:48:11 all (51/219): syscall4.sh WARNING pid 55749 (syscall4): ioctl sign-extension ioctl b2f081bc84958cba WARNING pid 55815 (syscall4): ioctl sign-extension ioctl 5182620302fa6d33 WARNING pid 55878 (syscall4): ioctl sign-extension ioctl 6d9425137c4b WARNING pid 55899 (syscall4): ioctl sign-extension ioctl 3ec13df44f74370c WARNING pid 56081 (syscall4): ioctl sign-extension ioctl 2db8b6bc68e7c3c6 WARNING pid 56160 (syscall4): ioctl sign-extension ioctl 389bde2da561 WARNING pid 56193 (syscall4): ioctl sign-extension ioctl 373e8a40c0f2 WARNING pid 56381 (syscall4): ioctl sign-extension ioctl 2e1637d342e4 WARNING pid 56383 (syscall4): ioctl sign-extension ioctl 4848c58d1ec9 WARNING pid 56690 (syscall4): ioctl sign-extension ioctl 733b07703d703f64 WARNING pid 56806 (syscall4): ioctl sign-extension ioctl 273217983ca8 WARNING pid 56856 (syscall4): ioctl sign-extension ioctl 7c1b86b2d15b WARNING pid 56877 (syscall4): ioctl sign-extension ioctl 3febe8a0fa49 WARNING pid 57007 (syscall4): ioctl sign-extension ioctl 71bc8127d40a0770 WARNING pid 57009 (syscall4): ioctl sign-extension ioctl 186182d4bd9b : : WARNING pid 65327 (syscall4): ioctl sign-extension ioctl 742e8f841d9 WARNING pid 65456 (syscall4): ioctl sign-extension ioctl 65ec4867df9d6303 WARNING pid 65463 (syscall4): ioctl sign-extension ioctl 7665e955f59605e1 WARNING pid 65516 (syscall4): ioctl sign-extension ioctl 54fed451b906 WARNING pid 65714 (syscall4): ioctl sign-extension ioctl 1a8546e4ef7a WARNING pid 65727 (syscall4): ioctl sign-extension ioctl b6504cda56b365f WARNING pid 65729 (syscall4): ioctl sign-extension ioctl 9238b0eff725bae0 WARNING pid 65754 (syscall4): ioctl sign-extension ioctl 325ecc4ed73c WARNING pid 65768 (syscall4): ioctl sign-extension ioctl 7f4924938477 WARNING pid 66039 (syscall4): ioctl sign-extension ioctl db2c4e03f339e349 WARNING pid 66194 (syscall4): ioctl sign-extension ioctl 29c1a6418ac52648 WARNING pid 66200 (syscall4): ioctl sign-extension ioctl 1d18c26521c73b36 WARNING pid 66229 (syscall4): ioctl sign-extension ioctl 617a4ff6ba4defec 20130822 05:18:18 all (52/219): nullfs12.sh 20130822 05:18:20 all (53/219): nfs13.sh 20130822 05:18:20 all (54/219): linger.sh 20130822 05:34:31 all (55/219): symlink2.sh 20130822 05:34:38 all (56/219): tmpfs.sh lock order reversal: 1st 0xfffff8070bc3da28 tmpfs (tmpfs) @ kern/vfs_vnops.c:699 2nd 0xfffffe0786027528 bufwait (bufwait) @ vm/vm_pager.c:308 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081f9a82c0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081f9a8370 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081f9a8400 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081f9a8530 initpbuf() at initpbuf+0xd1/frame 0xfffffe081f9a8560 getpbuf() at getpbuf+0x10d/frame 0xfffffe081f9a8590 swap_pager_getpages() at swap_pager_getpages+0x27f/frame 0xfffffe081f9a8620 uiomove_object() at uiomove_object+0x146/frame 0xfffffe081f9a8690 VOP_READ_APV() at VOP_READ_APV+0x108/frame 0xfffffe081f9a86c0 vn_read() at vn_read+0x244/frame 0xfffffe081f9a8750 vn_io_fault() at vn_io_fault+0x154/frame 0xfffffe081f9a88d0 dofileread() at dofileread+0x7b/frame 0xfffffe081f9a8910 kern_readv() at kern_readv+0x48/frame 0xfffffe081f9a8950 sys_read() at sys_read+0x63/frame 0xfffffe081f9a89a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081f9a8ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081f9a8ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b5737a, rsp = 0x7fffffffc538, rbp = 0x7fffffffd5f0 --- Aug 22 05:39:48 t2 kernel: pid 68522 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68559 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68533 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68538 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68541 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68548 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68526 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68528 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:50 t2 kernel: pid 68512 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:51 t2 kernel: pid 68524 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:51 t2 kernel: pid 68519 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:51 t2 kernel: pid 68543 (symlink), uid 0, was killed: exceeded maximum CPU limit Aug 22 05:39:51 t2 kernel: pid 68553 (symlink), uid 0, was killed: exceeded maximum CPU limit 20130822 05:45:24 all (57/219): quota9.sh 20130822 05:45:28 all (58/219): tmpfs9.sh 20130822 05:57:22 all (59/219): union.sh 20130822 06:09:49 all (60/219): nullfs.sh 20130822 06:14:34 all (61/219): extattrctl.sh lock order reversal: 1st 0xfffff804897a7438 ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xfffff8077327cc98 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1710 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081f0b30f0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081f0b31a0 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081f0b3230 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081f0b3360 ffs_vgetf() at ffs_vgetf+0x143/frame 0xfffffe081f0b33f0 ufs_root() at ufs_root+0x4b/frame 0xfffffe081f0b3430 ufs_extattr_autostart() at ufs_extattr_autostart+0x79/frame 0xfffffe081f0b3490 ffs_mount() at ffs_mount+0x1e98/frame 0xfffffe081f0b36b0 vfs_donmount() at vfs_donmount+0xc84/frame 0xfffffe081f0b3960 sys_nmount() at sys_nmount+0x72/frame 0xfffffe081f0b39a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081f0b3ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081f0b3ab0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a989ca, rsp = 0x7fffffffc728, rbp = 0x7fffffffcc90 --- 20130822 06:14:35 all (62/219): jail4.sh 20130822 06:14:36 all (63/219): tmpfs3.sh 20130822 06:27:02 all (64/219): suj.sh ffs_fsync: dirty 0xfffff804f1091c30: tag ufs, type VREG usecount 1, writecount 1, refcount 333 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff805ed446100 ref 0 pages 2648 lock type ufs: EXCL by thread 0xfffff8078ff4f000 (pid 74072, rw, tid 100772) #0 0xffffffff8089e9c9 at __lockmgr_args+0x9a9 #1 0xffffffff80b122b2 at ffs_lock+0x92 #2 0xffffffff80d68fd5 at VOP_LOCK1_APV+0xf5 #3 0xffffffff80976406 at _vn_lock+0xc6 #4 0xffffffff8097871a at vn_write+0x1da #5 0xffffffff80974cfa at vn_io_fault+0x22a #6 0xffffffff80911125 at dofilewrite+0x85 #7 0xffffffff80910e48 at kern_writev+0x48 #8 0xffffffff80910df3 at sys_write+0x63 #9 0xffffffff80cbe202 at amd64_syscall+0x282 #10 0xffffffff80ca14cb at Xfast_syscall+0xfb ino 32772, on dev md5a 20130822 06:47:08 all (65/219): quota5.sh 20130822 06:47:08 all (66/219): posix_fadvise.sh 20130822 06:47:10 all (67/219): vnodes.sh 20130822 06:48:20 all (68/219): sndstat.sh 20130822 06:48:20 all (69/219): nfs3.sh 20130822 06:49:12 all (70/219): nullfs11.sh 20130822 06:51:40 all (71/219): msdos.sh 20130822 07:01:44 all (72/219): suj6.sh 20130822 07:02:47 all (73/219): snap2-1.sh 20130822 07:02:53 all (74/219): trim2.sh 20130822 07:02:55 all (75/219): sendfile4.sh 20130822 07:02:56 all (76/219): alternativeFlushPath.sh 20130822 07:23:32 all (77/219): quota1.sh 20130822 07:33:54 all (78/219): procfs2.sh 20130822 07:34:00 all (79/219): procfs4.sh 20130822 07:44:42 all (80/219): nullfs9.sh lock order reversal: 1st 0xfffff8074d5152d8 tmpfs (tmpfs) @ kern/vfs_subr.c:2099 2nd 0xfffff80696355a28 null (null) @ /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:748 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081f1d02a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081f1d0350 witness_checkorder() at witness_checkorder+0xd4f/frame 0xfffffe081f1d03e0 __lockmgr_args() at __lockmgr_args+0x6f2/frame 0xfffffe081f1d0510 null_reclaim() at null_reclaim+0x7a/frame 0xfffffe081f1d0580 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x10e/frame 0xfffffe081f1d05b0 vgonel() at vgonel+0x222/frame 0xfffffe081f1d0620 vrecycle() at vrecycle+0x4a/frame 0xfffffe081f1d0650 null_inactive() at null_inactive+0x57/frame 0xfffffe081f1d0670 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x10e/frame 0xfffffe081f1d06a0 vinactive() at vinactive+0x157/frame 0xfffffe081f1d06f0 vputx() at vputx+0x370/frame 0xfffffe081f1d0750 nullfs_unlink_lowervp() at nullfs_unlink_lowervp+0x31/frame 0xfffffe081f1d0790 vfs_notify_upper() at vfs_notify_upper+0x209/frame 0xfffffe081f1d07f0 kern_unlinkat() at kern_unlinkat+0x1f4/frame 0xfffffe081f1d09a0 amd64_syscall() at amd64_syscall+0x282/frame 0xfffffe081f1d0ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081f1d0ab0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80093169a, rsp = 0x7fffffffd5e8, rbp = 0x7fffffffd680 --- 20130822 07:44:43 all (81/219): suj3.sh cryptosoft0: on motherboard GEOM_ELI: Device md5.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software GEOM_ELI: md5 has been killed. GEOM_ELI: Device md5.eli destroyed. 20130822 07:50:59 all (82/219): umountf6.sh 20130822 07:54:30 all (83/219): suj5.sh 20130822 07:55:37 all (84/219): parallelmount.sh Aug 22 07:56:10 t2 last message repeated 5 times Aug 22 07:57:11 t2 last message repeated 8 times Aug 22 07:57:54 t2 last message repeated 4 times Aug 22 07:57:59 t2 last message repeated 2 times 20130822 07:58:03 all (85/219): procfs3.sh 20130822 07:58:16 all (86/219): namecache2.sh 20130822 08:00:17 all (87/219): radix.sh 20130822 08:02:56 all (88/219): symlink.sh 20130822 08:19:48 all (89/219): marcus2.sh witness_lock_list_get: witness exhausted 20130822 08:41:41 all (90/219): trim5.sh 20130822 08:41:42 all (91/219): umountf4.sh Aug 22 08:42:07 t2 kernel: pid 75105 (sh), uid 1004, was killed: exceeded maximum CPU limit 20130822 08:46:22 all (92/219): umountf2.sh 20130822 08:53:42 all (93/219): jail2.sh 20130822 08:53:42 all (94/219): jail.sh 20130822 08:53:43 all (95/219): linger4.sh 20130822 09:10:44 all (96/219): kevent2.sh 20130822 09:10:49 all (97/219): truncate.sh 20130822 09:40:17 all (98/219): mmap3.sh 20130822 09:50:21 all (99/219): fpclone.sh 20130822 09:52:23 all (100/219): nullfs13.sh 20130822 09:52:24 all (101/219): extattr.sh 20130822 09:52:29 all (102/219): quota10.sh Aug 22 09:58:46 t2 last message repeated 2 times 20130822 10:24:25 all (103/219): suj8.sh Aug 22 10:24:26 t2 kernel: pid 76696 (dd), uid 0 inumber 4 on /mnt: filesystem full Aug 22 10:24:27 t2 kernel: pid 76717 (dd), uid 0 inumber 5 on /mnt: filesystem full 20130822 10:24:29 all (104/219): umountf.sh 20130822 10:26:55 all (105/219): ffs_syncvnode.sh 20130822 10:57:36 all (106/219): truncate3.sh 20130822 10:57:43 all (107/219): nfs8.sh 20130822 10:57:45 all (108/219): suj4.sh 20130822 11:27:50 all (109/219): sendfile3.sh 20130822 11:28:57 all (110/219): thr.sh 20130822 11:29:07 all (111/219): pts.sh 20130822 11:34:44 all (112/219): lockf.sh Aug 22 11:34:45 t2 mountd[883]: bad exports list line /mnt 20130822 11:34:53 all (113/219): suj9.sh interrupt total irq3: uart1 83078 irq9: acpi0 2 irq20: ehci1 135940 irq22: ehci0 60486 cpu0:timer 41116979 irq264: isci0 11665191 irq266: hdac0 104 irq267: igb0:que 0 51565 irq268: igb0:que 1 51597 irq269: igb0:que 2 124621 irq270: igb0:que 3 51267 irq271: igb0:que 4 51819 irq272: igb0:que 5 156623 irq273: igb0:que 6 51623 irq274: igb0:que 7 51707 irq275: igb0:link 1 irq286: ahci0 248 irq287: hdac1 4 cpu1:timer 43488193 cpu4:timer 42927989 cpu17:timer 45152378 cpu8:timer 43490954 cpu9:timer 43319004 cpu20:timer 45068633 cpu10:timer 43861351 cpu13:timer 44313931 cpu11:timer 44651642 cpu21:timer 44650630 cpu7:timer 43116646 cpu19:timer 44686418 cpu3:timer 43084687 cpu16:timer 44375266 cpu5:timer 43274845 cpu15:timer 45061898 cpu6:timer 43014385 cpu14:timer 44440820 cpu2:timer 42852636 cpu18:timer 45069016 cpu12:timer 44727589 cpu22:timer 44606954 cpu23:timer 44774270 Total 1067662992 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07fc5fc630 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07fc5fc6e0 watchdog_fire() at watchdog_fire+0x8d/frame 0xfffffe07fc5fc710 hardclock_cnt() at hardclock_cnt+0x297/frame 0xfffffe07fc5fc780 handleevents() at handleevents+0xbe/frame 0xfffffe07fc5fc7d0 timercb() at timercb+0x208/frame 0xfffffe07fc5fc830 lapic_handle_timer() at lapic_handle_timer+0x9c/frame 0xfffffe07fc5fc860 Xtimerint() at Xtimerint+0x8c/frame 0xfffffe07fc5fc860 --- interrupt, rip = 0xffffffff80c9c356, rsp = 0xfffffe07fc5fc920, rbp = 0xfffffe07fc5fc930 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe07fc5fc930 acpi_cpu_idle() at acpi_cpu_idle+0x13b/frame 0xfffffe07fc5fc980 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe07fc5fc9a0 cpu_idle() at cpu_idle+0x93/frame 0xfffffe07fc5fc9c0 sched_idletd() at sched_idletd+0x1e8/frame 0xfffffe07fc5fca70 fork_exit() at fork_exit+0x84/frame 0xfffffe07fc5fcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07fc5fcab0 --- trap 0, rip = 0, rsp = 0xfffffe07fc5fcb70, rbp = 0 --- KDB: enter: watchdog timeout panic: mi_switch: switch in a critical section cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe07fc5fc080 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe07fc5fc130 vpanic() at vpanic+0x126/frame 0xfffffe07fc5fc170 kassert_panic() at kassert_panic+0x136/frame 0xfffffe07fc5fc1e0 mi_switch() at mi_switch+0xa4/frame 0xfffffe07fc5fc220 turnstile_wait() at turnstile_wait+0x3af/frame 0xfffffe07fc5fc270 __mtx_lock_sleep() at __mtx_lock_sleep+0x1fe/frame 0xfffffe07fc5fc2f0 __mtx_lock_flags() at __mtx_lock_flags+0xf5/frame 0xfffffe07fc5fc340 ukbd_poll() at ukbd_poll+0x2c/frame 0xfffffe07fc5fc370 kbdmux_poll() at kbdmux_poll+0x5b/frame 0xfffffe07fc5fc390 cngrab() at cngrab+0x35/frame 0xfffffe07fc5fc3b0 kdb_trap() at kdb_trap+0x124/frame 0xfffffe07fc5fc410 trap() at trap+0x77b/frame 0xfffffe07fc5fc630 calltrap() at calltrap+0x8/frame 0xfffffe07fc5fc630 --- trap 0x3, rip = 0xffffffff808f18be, rsp = 0xfffffe07fc5fc6f0, rbp = 0xfffffe07fc5fc710 --- kdb_enter() at kdb_enter+0x3e/frame 0xfffffe07fc5fc710 hardclock_cnt() at hardclock_cnt+0x297/frame 0xfffffe07fc5fc780 handleevents() at handleevents+0xbe/frame 0xfffffe07fc5fc7d0 timercb() at timercb+0x208/frame 0xfffffe07fc5fc830 lapic_handle_timer() at lapic_handle_timer+0x9c/frame 0xfffffe07fc5fc860 Xtimerint() at Xtimerint+0x8c/frame 0xfffffe07fc5fc860 --- interrupt, rip = 0xffffffff80c9c356, rsp = 0xfffffe07fc5fc920, rbp = 0xfffffe07fc5fc930 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe07fc5fc930 acpi_cpu_idle() at acpi_cpu_idle+0x13b/frame 0xfffffe07fc5fc980 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe07fc5fc9a0 cpu_idle() at cpu_idle+0x93/frame 0xfffffe07fc5fc9c0 sched_idletd() at sched_idletd+0x1e8/frame 0xfffffe07fc5fca70 fork_exit() at fork_exit+0x84/frame 0xfffffe07fc5fcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe07fc5fcab0 --- trap 0, rip = 0, rsp = 0xfffffe07fc5fcb70, rbp = 0 --- Uptime: 14h19m30s Dumping 2090 out of 31644 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete $ ps -N kernel.debug -M /var/crash/vmcore.40 -auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 0 0,0 0,0 0 0 - DLs 1jan70 145:37,62 [kernel] 0 0 0 -100 0 - root 1 0,0 0,0 9344 0 - DLs 1jan70 0:04,80 [init] 0 0 0 20 0 wait root 2 0,0 0,0 0 0 - DL 1jan70 0:00,00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 3 0,0 0,0 0 0 - DL 1jan70 0:00,00 [xpt_thrd] 0 0 0 -16 0 ccb_scan root 4 0,0 0,0 0 0 - DL 1jan70 0:00,21 [enc_daemon0] 0 0 0 -16 0 idle root 5 0,0 0,0 0 0 - DL 1jan70 107:04,31 [pagedaemon] 0 0 0 -16 0 psleep root 6 0,0 0,0 0 0 - DL 1jan70 3:13,07 [vmdaemon] 0 0 0 -16 0 psleep root 7 0,0 0,0 0 0 - DL 1jan70 0:00,00 [pagezero] 0 0 0 155 0 pgzero root 8 0,0 0,0 0 0 - DL 1jan70 6:38,20 [bufdaemon] 0 0 0 34 0 psleep root 9 0,0 0,0 0 0 - DL 1jan70 10:57,64 [syncer] 0 0 0 16 0 syncer root 10 0,0 0,0 0 0 - DL 1jan70 0:00,00 [audit] 0 0 0 -16 0 audit_wo root 11 0,0 0,0 0 0 - RL 1jan70 12532:19,67 [idle] 0 0 0 155 0 - root 12 0,0 0,0 0 0 - WL 1jan70 9:46,27 [intr] 0 0 0 -76 0 - root 13 0,0 0,0 0 0 - DL 1jan70 80:36,25 [geom] 0 0 0 -8 0 - root 14 0,0 0,0 0 0 - DL 1jan70 0:03,25 [yarrow] 0 0 0 -16 0 - root 15 0,0 0,0 0 0 - DL 1jan70 0:18,87 [usb] 0 0 0 -68 0 - root 16 0,0 0,0 0 0 - DL 1jan70 1:04,11 [vnlru] 0 0 0 20 0 vlruwt root 17 0,0 0,0 0 0 - DL 1jan70 40:10,44 [softdepflush] 0 0 0 20 0 sdflush root 683 0,0 0,0 16504 0 - Ds 1jan70 0:00,00 [moused] 0 1 0 52 0 select root 700 0,0 0,0 13120 0 - Ds 1jan70 4:18,50 [devd] 0 1 0 20 0 select root 829 0,0 0,0 14300 0 - Ds 1jan70 2:50,81 [syslogd] 0 1 0 20 0 select root 842 0,0 0,0 16388 0 - Ds 1jan70 0:01,08 [rpcbind] 0 1 0 20 0 select root 872 0,0 0,0 12532 0 - DWs - 0:00,00 [nfsuserd] 0 1 0 52 0 pause root 873 0,0 0,0 12532 0 - D 1jan70 0:00,86 [nfsuserd] 0 872 0 20 0 select root 874 0,0 0,0 12532 0 - D 1jan70 0:01,44 [nfsuserd] 0 872 0 20 0 select root 875 0,0 0,0 12532 0 - D 1jan70 0:00,65 [nfsuserd] 0 872 0 20 0 select root 876 0,0 0,0 12532 0 - D 1jan70 0:00,29 [nfsuserd] 0 872 0 20 0 select root 883 0,0 0,0 26596 0 - Ds 1jan70 4:17,17 [mountd] 0 1 0 20 0 select root 889 0,0 0,0 20340 0 - Ds 1jan70 0:00,07 [nfsd] 0 1 0 20 0 select root 892 0,0 0,0 12148 0 - D 1jan70 1:37,65 [nfsd] 0 889 0 20 0 rpcsvc root 929 0,0 0,0 25188 0 - Ds 1jan70 0:07,96 [ntpd] 0 1 0 20 0 select root 932 0,0 0,0 14276 0 - Rs 1jan70 2:39,31 [powerd] 0 1 0 20 0 - root 960 0,0 0,0 56244 0 - Ds 1jan70 0:00,01 [sshd] 0 1 0 20 0 select root 963 0,0 0,0 23856 0 - Ds 1jan70 0:04,49 [sendmail] 0 1 0 20 0 select smmsp 966 0,0 0,0 23856 0 - Ds 1jan70 0:00,70 [sendmail] 25 1 0 20 0 pause root 970 0,0 0,0 16392 0 - Ds 1jan70 0:01,18 [cron] 0 1 0 20 0 nanslp root 979 0,0 0,0 81512 0 - Ds 1jan70 0:00,03 [sshd] 0 960 0 52 0 select root 983 0,0 0,0 6128 2040 - Ds 1jan70 1:28,98 [watchdogd] 0 1 0 -52 0 wait root 1052 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,19 [getty] 0 1 0 20 0 ttyin root 1053 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1054 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1055 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1056 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1057 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1058 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1059 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,01 [getty] 0 1 0 52 0 ttyin root 1060 0,0 0,0 14296 0 - Ds+ 1jan70 0:00,00 [getty] 0 1 0 52 0 ttyin pho 1061 0,0 0,0 81512 0 - D 1jan70 0:04,20 [sshd] 1001 979 0 20 0 select pho 1062 0,0 0,0 20396 0 - DWs - 0:00,00 [bash] 1001 1061 0 21 0 wait root 1075 0,0 0,0 47524 0 - DW - 0:00,00 [su] 0 1062 0 20 0 wait root 1076 0,0 0,0 20244 0 - DW - 0:00,00 [csh] 0 1075 0 21 0 pause root 1078 0,0 0,0 20396 0 - DW - 0:00,00 [bash] 0 1076 0 25 0 wait root 1081 0,0 0,0 31932 0 - L+ 1jan70 29:14,96 [top] 0 1078 0 20 0 Giant root 1082 0,0 0,0 81512 0 - Ds 1jan70 0:00,08 [sshd] 0 960 0 30 0 select pho 1084 0,0 0,0 81512 0 - D 1jan70 0:02,17 [sshd] 1001 1082 0 20 0 select pho 1085 0,0 0,0 20396 0 - DWs - 0:00,00 [bash] 1001 1084 0 24 0 wait root 1089 0,0 0,0 47524 0 - DW - 0:00,00 [su] 0 1085 0 21 0 wait root 1090 0,0 0,0 20244 0 - DW - 0:00,00 [csh] 0 1089 0 23 0 pause root 1092 0,0 0,0 20396 0 - DW - 0:00,00 [bash] 0 1090 0 23 0 wait root 1095 0,0 0,0 16864 0 - DW+ - 0:00,00 [sh] 0 1092 0 52 0 wait root 14901 0,0 0,0 0 0 - DLs 1jan70 0:00,03 [aiod1] 0 1 0 -16 0 aiordy root 14902 0,0 0,0 0 0 - DLs 1jan70 0:00,02 [aiod2] 0 1 0 -16 0 aiordy root 14903 0,0 0,0 0 0 - DLs 1jan70 0:00,02 [aiod3] 0 1 0 -16 0 aiordy root 14904 0,0 0,0 0 0 - DLs 1jan70 0:00,03 [aiod4] 0 1 0 -16 0 aiordy root 30890 0,0 0,0 0 0 - DL 1jan70 10:39,99 [zfskern] 0 0 0 -8 0 ffffffff81bb15f0 root 75888 0,0 0,0 0 0 - DL 1jan70 0:00,00 [crypto] 0 0 0 -16 0 crypto_w root 75889 0,0 0,0 0 0 - DL 1jan70 0:00,00 [crypto returns] 0 0 0 -16 0 crypto_r root 85507 0,0 0,0 16864 0 - DW+ - 0:00,00 [sh] 0 1095 0 24 0 wait root 85520 0,0 0,0 0 0 - DL 1jan70 4:03,16 [md5] 0 0 0 -8 0 mdwait root 88416 0,0 0,0 47508 0 - DW+ - 0:00,00 [su] 0 85507 0 24 0 wait stress 88417 0,0 0,0 16864 0 - DW - 0:00,00 [sh] 1004 88416 0 52 0 wait stress 88422 0,0 0,0 14276 0 - D 1jan70 0:00,01 [run] 1004 88417 0 20 0 wait stress 88423 0,0 0,0 14276 0 - D 1jan70 0:00,00 [run] 1004 88422 0 20 0 wait stress 88424 0,0 0,0 14276 0 - D 1jan70 0:00,01 [run] 1004 88423 0 20 0 wait stress 88546 0,0 0,0 14276 0 - D 1jan70 0:00,01 [lockf2] 1004 88424 0 20 0 wait stress 88547 0,0 0,0 0 0 - ZW - 0:00,00 1004 88424 0 -100 0 - stress 88548 0,0 0,0 0 0 - ZW - 0:00,00 1004 88424 0 -100 0 - stress 88549 0,0 0,0 14280 0 - D 1jan70 0:00,01 [rw] 1004 88424 0 20 0 wait stress 88550 0,0 0,0 14272 0 - D 1jan70 0:00,01 [fts] 1004 88424 0 20 0 wait stress 88551 0,0 0,0 14280 0 - D 1jan70 0:00,01 [link] 1004 88424 0 20 0 wait stress 88552 0,0 0,0 14280 0 - D 1jan70 0:00,01 [lockf] 1004 88424 0 20 0 wait stress 88553 0,0 0,0 14280 0 - D 1jan70 0:00,01 [creat] 1004 88424 0 20 0 wait stress 88554 0,0 0,0 14280 0 - D 1jan70 0:00,01 [mkdir] 1004 88424 0 20 0 wait stress 88555 0,0 0,0 0 0 - ZW - 0:00,00 1004 88424 0 -100 0 - stress 88556 0,0 0,0 0 0 - ZW - 0:00,00 1004 88424 0 -100 0 - stress 88557 0,0 0,0 14280 0 - D 1jan70 0:00,01 [mkfifo] 1004 88424 0 20 0 wait stress 88558 0,0 0,0 14276 0 - D 1jan70 0:00,01 [lockf2] 1004 88546 0 20 0 wait stress 88559 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88560 0,0 0,0 14280 0 - D 1jan70 0:00,01 [rw] 1004 88549 0 20 0 wait stress 88561 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88562 0,0 0,0 14280 0 - D 1jan70 0:05,47 [rw] 1004 88560 0 20 0 suspfs stress 88563 0,0 0,0 14280 0 - D 1jan70 0:00,01 [link] 1004 88551 0 20 0 wait stress 88564 0,0 0,0 14272 0 - D 1jan70 0:00,01 [fts] 1004 88550 0 20 0 wait stress 88565 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88566 0,0 0,0 14280 0 - D 1jan70 0:07,82 [rw] 1004 88560 0 20 0 suspfs stress 88567 0,0 0,0 14272 0 - D 1jan70 0:01,60 [fts] 1004 88564 0 20 0 suspfs stress 88568 0,0 0,0 14280 0 - D 1jan70 0:05,29 [link] 1004 88563 0 20 0 suspfs stress 88569 0,0 0,0 14280 0 - D 1jan70 0:04,84 [rw] 1004 88560 0 20 0 suspfs stress 88570 0,0 0,0 14272 0 - D 1jan70 0:00,48 [fts] 1004 88564 0 20 0 suspfs stress 88571 0,0 0,0 14280 0 - D 1jan70 0:05,29 [link] 1004 88563 0 20 0 suspfs stress 88572 0,0 0,0 14280 0 - D 1jan70 0:07,21 [rw] 1004 88560 0 20 0 suspfs stress 88573 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88574 0,0 0,0 14280 0 - D 1jan70 0:00,01 [creat] 1004 88553 0 20 0 wait stress 88575 0,0 0,0 14272 0 - D 1jan70 0:00,56 [fts] 1004 88564 0 20 0 suspfs stress 88576 0,0 0,0 14280 0 - D 1jan70 0:04,47 [link] 1004 88563 0 20 0 suspfs stress 88577 0,0 0,0 14280 0 - D 1jan70 0:05,55 [rw] 1004 88560 0 20 0 suspfs stress 88578 0,0 0,0 14272 0 - D 1jan70 0:00,51 [fts] 1004 88564 0 20 0 suspfs stress 88579 0,0 0,0 14280 0 - D 1jan70 0:06,46 [link] 1004 88563 0 20 0 suspfs stress 88580 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88581 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 22 0 suspfs stress 88582 0,0 0,0 14280 0 - D 1jan70 0:06,41 [rw] 1004 88560 0 20 0 suspfs stress 88583 0,0 0,0 14272 0 - D 1jan70 0:00,52 [fts] 1004 88564 0 20 0 suspfs stress 88584 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88585 0,0 0,0 14280 0 - D 1jan70 0:06,52 [link] 1004 88563 0 20 0 suspfs stress 88586 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 23 0 suspfs stress 88587 0,0 0,0 14280 0 - D 1jan70 0:04,54 [rw] 1004 88560 0 20 0 suspfs stress 88588 0,0 0,0 14272 0 - D 1jan70 0:00,16 [fts] 1004 88564 0 20 0 suspfs stress 88589 0,0 0,0 14280 0 - D 1jan70 0:05,31 [link] 1004 88563 0 20 0 suspfs stress 88590 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 23 0 suspfs stress 88591 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88592 0,0 0,0 14280 0 - D 1jan70 0:04,07 [rw] 1004 88560 0 20 0 suspfs stress 88593 0,0 0,0 14272 0 - D 1jan70 0:00,74 [fts] 1004 88564 0 20 0 suspfs stress 88594 0,0 0,0 14280 0 - D 1jan70 0:03,38 [link] 1004 88563 0 20 0 suspfs stress 88595 0,0 0,0 14280 0 - D 1jan70 0:04,71 [creat] 1004 88574 0 20 0 suspfs stress 88596 0,0 0,0 14280 0 - D 1jan70 0:05,92 [rw] 1004 88560 0 20 0 suspfs stress 88597 0,0 0,0 14272 0 - D 1jan70 0:00,25 [fts] 1004 88564 0 20 0 suspfs stress 88598 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88599 0,0 0,0 14280 0 - D 1jan70 0:03,26 [link] 1004 88563 0 20 0 suspfs stress 88600 0,0 0,0 14280 0 - D 1jan70 0:00,00 [mkdir] 1004 88554 0 20 0 wait stress 88601 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 22 0 suspfs stress 88602 0,0 0,0 14280 0 - D 1jan70 0:08,26 [rw] 1004 88560 0 20 0 suspfs stress 88603 0,0 0,0 14272 0 - DW - 0:00,00 [fts] 1004 88564 0 20 0 suspfs stress 88604 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88605 0,0 0,0 14280 0 - D 1jan70 0:04,98 [link] 1004 88563 0 20 0 suspfs stress 88606 0,0 0,0 14408 0 - D 1jan70 0:05,56 [mkdir] 1004 88600 0 20 0 suspfs stress 88607 0,0 0,0 14280 0 - D 1jan70 0:06,34 [creat] 1004 88574 0 20 0 suspfs stress 88608 0,0 0,0 14280 0 - D 1jan70 0:06,23 [link] 1004 88563 0 20 0 suspfs stress 88609 0,0 0,0 14280 0 - D 1jan70 0:06,28 [rw] 1004 88560 0 20 0 suspfs stress 88610 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88611 0,0 0,0 14272 0 - D 1jan70 0:00,26 [fts] 1004 88564 0 20 0 suspfs stress 88612 0,0 0,0 14408 0 - D 1jan70 0:02,31 [mkdir] 1004 88600 0 20 0 suspfs stress 88613 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 25 0 suspfs stress 88614 0,0 0,0 14280 0 - D 1jan70 0:03,19 [link] 1004 88563 0 20 0 suspfs stress 88615 0,0 0,0 14272 0 - D 1jan70 0:00,40 [fts] 1004 88564 0 20 0 suspfs stress 88616 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88617 0,0 0,0 14408 0 - D 1jan70 0:03,35 [mkdir] 1004 88600 0 20 0 suspfs stress 88618 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 25 0 suspfs stress 88619 0,0 0,0 14280 0 - D 1jan70 0:07,70 [link] 1004 88563 0 20 0 suspfs stress 88620 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88621 0,0 0,0 14408 0 - D 1jan70 0:04,53 [mkdir] 1004 88600 0 20 0 suspfs stress 88622 0,0 0,0 14280 0 - D 1jan70 0:04,42 [link] 1004 88563 0 20 0 suspfs stress 88623 0,0 0,0 14276 0 - DW - 0:00,00 [lockf2] 1004 88558 0 52 0 suspfs stress 88624 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 24 0 suspfs stress 88625 0,0 0,0 14408 0 - D 1jan70 0:02,80 [mkdir] 1004 88600 0 20 0 suspfs stress 88626 0,0 0,0 14280 0 - D 1jan70 0:04,45 [creat] 1004 88574 0 20 0 suspfs stress 88628 0,0 0,0 14280 0 - D 1jan70 0:05,31 [creat] 1004 88574 0 20 0 suspfs stress 88630 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 23 0 suspfs stress 88633 0,0 0,0 14280 0 - D 1jan70 0:05,37 [creat] 1004 88574 0 20 0 suspfs stress 88635 0,0 0,0 14280 0 - D 1jan70 0:04,96 [creat] 1004 88574 0 20 0 suspfs stress 88638 0,0 0,0 14280 0 - D 1jan70 0:07,13 [creat] 1004 88574 0 20 0 suspfs stress 88639 0,0 0,0 14280 0 - D 1jan70 0:03,34 [creat] 1004 88574 0 20 0 suspfs stress 88642 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 24 0 suspfs stress 88644 0,0 0,0 14280 0 - DW - 0:00,00 [creat] 1004 88574 0 25 0 suspfs stress 88648 0,0 0,0 14280 0 - D 1jan70 0:00,00 [mkfifo] 1004 88557 0 20 0 wait stress 88650 0,0 0,0 14280 0 - D 1jan70 0:00,01 [lockf] 1004 88552 0 20 0 wait stress 88651 0,0 0,0 14280 0 - D 1jan70 0:00,22 [mkfifo] 1004 88648 0 20 0 suspfs stress 88652 0,0 0,0 14280 0 - D 1jan70 0:00,07 [lockf] 1004 88650 0 20 0 suspfs stress 88653 0,0 0,0 14280 0 - D 1jan70 0:00,12 [mkfifo] 1004 88648 0 20 0 suspfs stress 88654 0,0 0,0 14280 0 - D 1jan70 0:01,43 [lockf] 1004 88650 0 20 0 suspfs stress 88655 0,0 0,0 14280 0 - D 1jan70 0:00,27 [mkfifo] 1004 88648 0 20 0 suspfs stress 88656 0,0 0,0 14280 0 - D 1jan70 0:02,69 [lockf] 1004 88650 0 20 0 suspfs stress 88657 0,0 0,0 0 0 - ZW - 0:00,00 1004 88648 0 -100 0 - stress 88658 0,0 0,0 14280 0 - D 1jan70 0:00,15 [mkfifo] 1004 88648 0 20 0 suspfs stress 88659 0,0 0,0 14280 0 - D 1jan70 0:00,05 [lockf] 1004 88650 0 20 0 suspfs stress 88660 0,0 0,0 14280 0 - D 1jan70 0:00,12 [mkfifo] 1004 88648 0 20 0 suspfs stress 88661 0,0 0,0 14280 0 - D 1jan70 0:00,01 [lockf] 1004 88650 0 20 0 suspfs stress 88662 0,0 0,0 0 0 - ZW - 0:00,00 1004 88648 0 -100 0 - stress 88663 0,0 0,0 14280 0 - D 1jan70 0:00,46 [lockf] 1004 88650 0 20 0 suspfs stress 88664 0,0 0,0 0 0 - ZW - 0:00,00 1004 88648 0 -100 0 - stress 88665 0,0 0,0 14280 0 - D 1jan70 0:00,05 [lockf] 1004 88650 0 20 0 suspfs stress 88666 0,0 0,0 14280 0 - D 1jan70 0:00,13 [mkfifo] 1004 88648 0 20 0 suspfs stress 88667 0,0 0,0 14280 0 - D 1jan70 0:00,31 [mkfifo] 1004 88648 0 20 0 suspfs stress 88668 0,0 0,0 14280 0 - D 1jan70 0:01,53 [lockf] 1004 88650 0 20 0 suspfs stress 88669 0,0 0,0 14280 0 - D 1jan70 0:00,15 [mkfifo] 1004 88648 0 20 0 suspfs stress 88670 0,0 0,0 14280 0 - D 1jan70 0:00,01 [lockf] 1004 88650 0 20 0 suspfs stress 88671 0,0 0,0 14280 0 - D 1jan70 0:00,06 [lockf] 1004 88650 0 20 0 suspfs stress 88672 0,0 0,0 14280 0 - D 1jan70 0:00,02 [lockf] 1004 88650 0 20 0 suspfs stress 88673 0,0 0,0 14280 0 - D 1jan70 0:01,83 [lockf] 1004 88650 0 20 0 suspfs stress 88674 0,0 0,0 14280 0 - D 1jan70 0:00,05 [lockf] 1004 88650 0 20 0 suspfs stress 88675 0,0 0,0 14280 0 - D 1jan70 0:02,62 [lockf] 1004 88650 0 20 0 suspfs stress 88676 0,0 0,0 14280 0 - D 1jan70 0:00,05 [lockf] 1004 88650 0 20 0 suspfs stress 88677 0,0 0,0 14280 0 - D 1jan70 0:00,02 [lockf] 1004 88650 0 20 0 suspfs stress 88678 0,0 0,0 14280 0 - D 1jan70 0:00,05 [lockf] 1004 88650 0 20 0 suspfs stress 88679 0,0 0,0 14280 0 - D 1jan70 0:01,31 [lockf] 1004 88650 0 20 0 suspfs stress 88680 0,0 0,0 14280 0 - D 1jan70 0:00,06 [lockf] 1004 88650 0 20 0 suspfs stress 89247 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88674 0 20 0 lockf stress 89248 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88680 0 20 0 lockf stress 89249 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88668 0 20 0 lockf stress 89250 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88671 0 20 0 lockf stress 89251 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88656 0 23 0 lockf stress 89252 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88677 0 20 0 lockf stress 89253 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88670 0 20 0 lockf stress 89254 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88672 0 20 0 lockf stress 89255 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88661 0 20 0 lockf stress 89256 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88676 0 20 0 lockf stress 89257 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88678 0 20 0 lockf stress 89258 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88675 0 20 0 lockf stress 89259 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88652 0 20 0 lockf stress 89260 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88663 0 20 0 lockf stress 89261 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88673 0 20 0 lockf stress 89262 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88659 0 20 0 lockf stress 89263 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88654 0 20 0 lockf stress 89264 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88679 0 20 0 lockf stress 89265 0,0 0,0 14280 0 - D 1jan70 0:00,00 [lockf] 1004 88665 0 20 0 lockf stress 89266 0,0 0,0 14280 0 - D 1jan70 0:00,01 [mkfifo] 1004 88660 0 21 0 suspfs stress 89267 0,0 0,0 14280 0 - D 1jan70 0:00,00 [mkfifo] 1004 88666 0 20 0 suspfs stress 89268 0,0 0,0 14280 0 - D 1jan70 0:00,02 [mkfifo] 1004 88669 0 23 0 suspfs stress 89269 0,0 0,0 14280 0 - D 1jan70 0:00,01 [mkfifo] 1004 88653 0 22 0 suspfs stress 89270 0,0 0,0 14280 0 - D 1jan70 0:00,02 [mkfifo] 1004 88667 0 24 0 suspfs stress 89271 0,0 0,0 14280 0 - D 1jan70 0:00,06 [mkfifo] 1004 88651 0 29 0 suspfs stress 89272 0,0 0,0 14280 0 - D 1jan70 0:00,02 [mkfifo] 1004 88658 0 23 0 suspfs stress 89273 0,0 0,0 14280 0 - D 1jan70 0:00,03 [mkfifo] 1004 88655 0 24 0 suspfs root 89274 0,0 0,0 16864 0 - D 1jan70 0:00,02 [sh] 0 983 0 -52 0 wait root 89275 0,0 0,0 16656 0 - D 1jan70 0:00,01 [ls] 0 89274 0 -52 0 suspfs $ (kgdb) proc 88562 [Switching to thread 240 (Thread 102639)]#0 sched_switch (td=0xfffff8001db8b490, newtd=, flags=) at ../../../kern/sched_ule.c:1954 1954 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xfffff8001db8b490, newtd=, flags=) at ../../../kern/sched_ule.c:1954 #1 0xffffffff808c44c9 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:493 #2 0xffffffff808fe685 in sleepq_switch (wchan=, pri=) at ../../../kern/subr_sleepqueue.c:541 #3 0xffffffff808feab4 in sleepq_catch_signals (wchan=, pri=0x77) at ../../../kern/subr_sleepqueue.c:429 #4 0xffffffff808fe71f in sleepq_wait_sig (wchan=0x0, pri=0x0) at ../../../kern/subr_sleepqueue.c:634 #5 0xffffffff808c3e00 in _sleep (ident=, lock=, priority=0x177, wmesg=0xffffffff80f98016 "suspfs", sbt=, pr=0x0) at ../../../kern/kern_synch.c:253 #6 0xffffffff809771e6 in vn_start_write_locked (mp=0xfffff80212026000, flags=0x101) at ../../../kern/vfs_vnops.c:1504 #7 0xffffffff80978664 in vn_write (fp=0xfffff80204d26500, uio=0xfffffe081fd44970, active_cred=0xfffff8062d8f3700, flags=, td=) at ../../../kern/vfs_vnops.c:1556 #8 0xffffffff80974cfa in vn_io_fault (fp=0xfffff80204d26500, uio=0xfffffe081fd44970, active_cred=0xfffff8062d8f3700, flags=0x0, td=0xfffff8001db8b490) at ../../../kern/vfs_vnops.c:986 #9 0xffffffff80911125 in dofilewrite (td=0xfffff8001db8b490, fd=0x4, fp=0xfffff80204d26500, auio=0xfffffe081fd44970, offset=, flags=0x0) at file.h:302 #10 0xffffffff80910e48 in kern_writev (td=0xfffff8001db8b490, fd=0x4, auio=0xfffffe081fd44970) at ../../../kern/sys_generic.c:460 #11 0xffffffff80910df3 in sys_write (td=, uap=) at ../../../kern/sys_generic.c:376 #12 0xffffffff80cbe202 in amd64_syscall (td=0xfffff8001db8b490, traced=0x0) at subr_syscall.c:134 #13 0xffffffff80ca14cb in Xfast_syscall () at ../../../amd64/amd64/exception.S:391 #14 0x0000000800b5735a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 6 #6 0xffffffff809771e6 in vn_start_write_locked (mp=0xfffff80212026000, flags=0x101) at ../../../kern/vfs_vnops.c:1504 1504 error = msleep(&mp->mnt_flag, MNT_MTX(mp), (kgdb) p *mp $1 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff80f6f185 "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffffe00006bdb00}, mtx_lock = 0x4}, mnt_gen = 0x100, mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffff80014c0a7b8}, mnt_op = 0xffffffff8132e040, mnt_vfc = 0xffffffff8132e128, mnt_vnodecovered = 0xfffff806afb6c4e0, mnt_syncer = 0xfffff802378ef750, mnt_ref = 0xb96, mnt_nvnodelist = { tqh_first = 0xfffff806cc395000, tqh_last = 0xfffff802d88b8500}, mnt_nvnodelistsize = 0xb2b, mnt_activevnodelist = {tqh_first = 0xfffff80696c224e0, tqh_last = 0xfffff806cc395158}, mnt_activevnodelistsize = 0x106, mnt_writeopcount = 0x0, mnt_kern_flag = 0x48802144, mnt_flag = 0x100001300, mnt_opt = 0xfffff801a1208da0, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x78, mnt_stat = { f_version = 0x20030518, f_type = 0x35, f_flags = 0x100001300, f_bsize = 0x1000, f_iosize = 0x8000, f_blocks = 0x7ba5, f_bfree = 0x6010, f_bavail = 0x562c, f_files = 0x3ffe, f_ffree = 0x34d4, f_syncwrites = 0x66c88, f_asyncwrites = 0x106da7, f_syncreads = 0x41f2, f_asyncreads = 0xbe0b, f_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = { 0x5215de86, 0x832b42bb}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/md5a", '\0' , f_mntonname = "/mnt", '\0' }, mnt_cred = 0xfffff805c7bec200, mnt_data = 0xfffff80711905000, mnt_time = 0x0, mnt_iosize_max = 0x20000, mnt_export = 0xfffff804d3db9c00, mnt_label = 0x0, mnt_hashseed = 0xb4f310df, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x4ff403, mnt_susp_owner = 0xfffff80014093920, mnt_gjprovider = 0x0, mnt_explock = {lock_object = { lo_name = 0xffffffff80f6f196 "explock", lo_flags = 0x6730000, lo_data = 0x0, lo_witness = 0xfffffe00006d6180}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x6, pcs = { 0xffffffff8089e9c9, 0xffffffff80957a1a, 0xffffffff8095e170, 0xffffffff8095cd82, 0xffffffff80cbe202, 0xffffffff80ca14cb, 0x0 }}}, mnt_upper_link = {tqe_next = 0xfffff80014be0b58, tqe_prev = 0xfffff80014be0780}, mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffff802120263b8}} (kgdb) p *(struct thread *)0xfffff80014093920 $1 = {td_lock = 0xffffffff813d53e8, td_proc = 0xfffff80014144000, td_plist = {tqe_next = 0x0, tqe_prev = 0xfffff80014144010}, td_runq = {tqe_next = 0x0, tqe_prev = 0xffffffff813aba08}, td_slpq = {tqe_next = 0x0, tqe_prev = 0xfffff80564132680}, td_lockq = {tqe_next = 0x0, tqe_prev = 0xfffffe078ad267c8}, td_hash = {le_next = 0x0, le_prev = 0xfffffe0000867930}, td_cpuset = 0xfffff800140875a0, td_sel = 0x0, td_sleepqueue = 0x0, td_turnstile = 0xfffff80429ebe6c0, td_rlqe = 0x0, td_umtxq = 0xfffff80014097200, td_tid = 0x18726, td_sigqueue = {sq_signals = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xfffff800140939d8}, sq_proc = 0xfffff80014144000, sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x4, td_inhibitors = 0x2, td_pflags = 0x240000, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0xffffffff8157f80c, td_wmesg = 0xffffffff80fc1973 "sdflush", td_lastcpu = 0x11, td_oncpu = 0xff, td_owepreempt = 0x0, td_tsqueue = 0xff, td_locks = 0x0, td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_stopsched = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0x0}, td_sleeplocks = 0xffffffff81522080, td_intr_nesting_level = 0x0, td_pinned = 0x0, td_ucred = 0xfffff8000a201000, td_estcpu = 0x0, td_slptick = 0x312e05c, td_blktick = 0x0, td_swvoltick = 0x312e05c, td_cow = 0x0, td_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0, ru_isrss = 0x0, ru_minflt = 0x0, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x38245, ru_oublock = 0x4e645, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x2a2f029, ru_nivcsw = 0x52c53}, td_rux = { rux_runtime = 0x45fc6bf6b9e, rux_uticks = 0x0, rux_sticks = 0x492f6, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x86e76e01, rux_tu = 0x86e76e01}, td_incruntime = 0x66b8a4, td_runtime = 0x45fc7262442, td_pticks = 0x492f6, td_sticks = 0x0, td_iticks = 0x0, td_uticks = 0x0, td_intrval = 0x0, td_oldsigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_generation = 0x2a81c7c, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x0}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = "softdepflush\000\000\000\000\000\000\000", td_fpop = 0x0, td_dbgflags = 0x0, td_dbgksi = { ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = { _timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}, ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0, td_dbg_forked = 0x0, td_vp_reserv = 0x0, td_no_sleeping = 0x0, td_dom_rr_idx = 0x0, td_sigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_rqindex = 0x1e, td_base_pri = 0x78, td_priority = 0x78, td_pri_class = 0x3, td_user_pri = 0x78, td_base_user_pri = 0x78, td_pcb = 0xfffffe081de8eb80, td_state = TDS_INHIBITED, td_retval = {0x0, 0x0}, td_slpcallout = {c_links = {le = {le_next = 0x0, le_prev = 0xfffffe0000ccf5d0}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0xfffffe0000ccf5d0}}, c_time = 0xc972ba5b3037, c_precision = 0xfffffed, c_arg = 0xfffff80014093920, c_func = 0xffffffff808fe160 , c_lock = 0x0, c_flags = 0x116, c_cpu = 0x11}, td_frame = 0xfffffe081de8eac0, td_kstack_obj = 0xfffff800130bc500, td_kstack = 0xfffffe081de8b000, td_kstack_pages = 0x4, td_critnest = 0x1, td_md = { md_spinlock_count = 0x1, md_saved_flags = 0x246, md_spurflt_addr = 0x0}, td_sched = 0xfffff80014093d88, td_ar = 0x0, td_lprof = {{lh_first = 0x0}, { lh_first = 0x0}}, td_dtrace = 0xfffff8001413c700, td_errno = 0x0, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0x0, td_rfppwait_p = 0x0, td_ma = 0x0, td_ma_cnt = 0x0} Current language: auto; currently minimal (kgdb) p *(struct ufsmount *)((mp)->mnt_data) $2 = {um_mountp = 0xfffff80212026000, um_dev = 0xfffff8006749b400, um_cp = 0xfffff801244bb400, um_bo = 0xfffff80182468168, um_devvp = 0xfffff80182468000, um_fstype = 0x2, um_fs = 0xfffff80417ebf000, um_extattr = { uepm_lock = {lock_object = {lo_name = 0xffffffff80fc3020 "ufs_extattr_sx", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0x0}, sx_lock = 0x1}, uepm_list = {lh_first = 0x0}, uepm_ucred = 0x0, uepm_flags = 0x1}, um_nindir = 0x1000, um_bptrtodb = 0x3, um_seqinc = 0x8, um_lock = {lock_object = {lo_name = 0xffffffff80fc22bb "FFS", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, um_fsckpid = 0x0, um_numindirdeps = 0x0, softdep_workitem_pending = {lh_first = 0x0}, softdep_worklist_tail = 0xfffff807119050b8, softdep_journal_pending = {lh_first = 0x0}, softdep_journal_tail = 0xfffff807119050c8, softdep_jblocks = 0xfffff8058c3b6700, softdep_unlinked = {tqh_first = 0xfffff806a1281000, tqh_last = 0xfffff8077f0fc430}, softdep_dirtycg = {lh_first = 0x0}, softdep_on_journal = 0x0, softdep_on_worklist = 0x0, softdep_deps = 0x9c2, softdep_accdeps = 0x76e68f, softdep_req = 0x0, um_quotas = {0x0, 0x0}, um_cred = {0x0, 0x0}, um_btime = {0x0, 0x0}, um_itime = {0x0, 0x0}, um_qflags = "\000", um_savedmaxfilesize = 0x0, um_candelete = 0x0, um_writesuspended = 0x0, um_balloc = 0xffffffff80aea600 , um_blkatoff = 0xffffffff80b0c5f0 , um_truncate = 0xffffffff80aec5f0 , um_update = 0xffffffff80aec260 , um_valloc = 0xffffffff80ae54a0 , um_vfree = 0xffffffff80ae6340 , um_ifree = 0xffffffff80b11040 , um_rdonly = 0xffffffff80aedfc0 , um_snapgone = 0xffffffff80af1870 } (kgdb) f 8 #8 0xffffffff80974cfa in vn_io_fault (fp=0xfffff80204d26500, uio=0xfffffe081fd44970, active_cred=0xfffff8062d8f3700, flags=0x0, td=0xfffff8001db8b490) at ../../../kern/vfs_vnops.c:986 986 error = doio(fp, uio, active_cred, flags | FOF_OFFSET, td); (kgdb) l 981 rl_cookie = vn_rangelock_wlock(vp, uio->uio_offset, 982 uio->uio_offset + uio->uio_resid); 983 } 984 985 save = vm_fault_disable_pagefaults(); 986 error = doio(fp, uio, active_cred, flags | FOF_OFFSET, td); 987 if (error != EFAULT) 988 goto out; 989 990 atomic_add_long(&vn_io_faults_cnt, 1); (kgdb) info loc short_uio = {uio_iov = 0xffffffff80878ff9, uio_iovcnt = 0x1fd448e0, uio_offset = 0xfffffe081fd448d8, uio_resid = 0x0, uio_segflg = UIO_USERSPACE, uio_rw = UIO_WRITE, uio_td = 0xfffff8001db8b490} short_iovec = {{iov_base = 0xfffffffffffffff, iov_len = 0x4}} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/amd64/amd64/pmap.c =================================================================== --- /usr/src/sys/amd64/amd64/pmap.c (revision 254608) +++ /usr/src/sys/amd64/amd64/pmap.c (working copy) @@ -1687,8 +1687,6 @@ pmap_pinit(pmap_t pmap) vm_page_t pml4pg; int i; - PMAP_LOCK_INIT(pmap); - /* * allocate the page directory page */ @@ -1959,9 +1957,6 @@ pmap_release(pmap_t pmap) KASSERT(vm_radix_is_empty(&pmap->pm_root), ("pmap_release: pmap has reserved page table page(s)")); - rw_wlock(&pvh_global_lock); - rw_wunlock(&pvh_global_lock); - m = PHYS_TO_VM_PAGE(pmap->pm_pml4[PML4PML4I] & PG_FRAME); for (i = 0; i < NKPML4E; i++) /* KVA */ @@ -1973,7 +1968,6 @@ pmap_release(pmap_t pmap) m->wire_count--; atomic_subtract_int(&cnt.v_wire_count, 1); vm_page_free_zero(m); - PMAP_LOCK_DESTROY(pmap); } static int @@ -4765,10 +4759,12 @@ pmap_remove_write(vm_page_t m) { struct md_page *pvh; pmap_t pmap; + struct rwlock *lock; pv_entry_t next_pv, pv; pd_entry_t *pde; pt_entry_t oldpte, *pte; vm_offset_t va; + int pvh_gen, md_gen; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_remove_write: page %p is not managed", m)); @@ -4781,23 +4777,51 @@ pmap_remove_write(vm_page_t m) VM_OBJECT_ASSERT_WLOCKED(m->object); if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) return; - rw_wlock(&pvh_global_lock); + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); +retry_pv_loop: + rw_wlock(lock); if ((m->flags & PG_FICTITIOUS) != 0) goto small_mappings; - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); + if (!PMAP_TRYLOCK(pmap)) { + pvh_gen = pvh->pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (pvh_gen != pvh->pv_gen) { + PMAP_UNLOCK(pmap); + rw_wunlock(lock); + goto retry_pv_loop; + } + } va = pv->pv_va; pde = pmap_pde(pmap, va); if ((*pde & PG_RW) != 0) - (void)pmap_demote_pde(pmap, pde, va); + (void)pmap_demote_pde_locked(pmap, pde, va, &lock); + KASSERT(lock == VM_PAGE_TO_PV_LIST_LOCK(m), + ("inconsistent pv lock %p %p for page %p", + lock, VM_PAGE_TO_PV_LIST_LOCK(m), m)); PMAP_UNLOCK(pmap); } small_mappings: TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); + if (!PMAP_TRYLOCK(pmap)) { + pvh_gen = pvh->pv_gen; + md_gen = m->md.pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (pvh_gen != pvh->pv_gen || + md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + rw_wunlock(lock); + goto retry_pv_loop; + } + } pde = pmap_pde(pmap, pv->pv_va); KASSERT((*pde & PG_PS) == 0, ("pmap_remove_write: found a 2mpage in page %p's pv list", @@ -4815,8 +4839,9 @@ retry: } PMAP_UNLOCK(pmap); } + rw_wunlock(lock); vm_page_aflag_clear(m, PGA_WRITEABLE); - rw_wunlock(&pvh_global_lock); + rw_runlock(&pvh_global_lock); } /* Index: /usr/src/sys/arm/arm/pmap-v6.c =================================================================== --- /usr/src/sys/arm/arm/pmap-v6.c (revision 254608) +++ /usr/src/sys/arm/arm/pmap-v6.c (working copy) @@ -1814,7 +1814,6 @@ pmap_release(pmap_t pmap) } pmap_free_l1(pmap); - PMAP_LOCK_DESTROY(pmap); dprintf("pmap_release()\n"); } @@ -3225,7 +3224,6 @@ pmap_pinit(pmap_t pmap) { PDEBUG(1, printf("pmap_pinit: pmap = %08x\n", (uint32_t) pmap)); - PMAP_LOCK_INIT(pmap); pmap_alloc_l1(pmap); bzero(pmap->pm_l2, sizeof(pmap->pm_l2)); Index: /usr/src/sys/arm/arm/pmap.c =================================================================== --- /usr/src/sys/arm/arm/pmap.c (revision 254608) +++ /usr/src/sys/arm/arm/pmap.c (working copy) @@ -2479,7 +2479,6 @@ pmap_release(pmap_t pmap) } pmap_free_l1(pmap); - PMAP_LOCK_DESTROY(pmap); dprintf("pmap_release()\n"); } @@ -3819,7 +3818,6 @@ pmap_pinit(pmap_t pmap) { PDEBUG(1, printf("pmap_pinit: pmap = %08x\n", (uint32_t) pmap)); - PMAP_LOCK_INIT(pmap); pmap_alloc_l1(pmap); bzero(pmap->pm_l2, sizeof(pmap->pm_l2)); Index: /usr/src/sys/i386/i386/pmap.c =================================================================== --- /usr/src/sys/i386/i386/pmap.c (revision 254608) +++ /usr/src/sys/i386/i386/pmap.c (working copy) @@ -1738,8 +1738,6 @@ pmap_pinit(pmap_t pmap) vm_paddr_t pa; int i; - PMAP_LOCK_INIT(pmap); - /* * No need to allocate page table space yet but we do need a valid * page directory table. @@ -2051,7 +2049,6 @@ pmap_release(pmap_t pmap) atomic_subtract_int(&cnt.v_wire_count, 1); vm_page_free_zero(m); } - PMAP_LOCK_DESTROY(pmap); } static int Index: /usr/src/sys/i386/xen/pmap.c =================================================================== --- /usr/src/sys/i386/xen/pmap.c (revision 254608) +++ /usr/src/sys/i386/xen/pmap.c (working copy) @@ -1452,8 +1452,6 @@ pmap_pinit(pmap_t pmap) mtx_lock(&createdelete_lock); #endif - PMAP_LOCK_INIT(pmap); - /* * No need to allocate page table space yet but we do need a valid * page directory table. @@ -1821,7 +1819,6 @@ pmap_release(pmap_t pmap) #ifdef PAE pmap_qremove((vm_offset_t)pmap->pm_pdpt, 1); #endif - PMAP_LOCK_DESTROY(pmap); #ifdef HAMFISTED_LOCKING mtx_unlock(&createdelete_lock); Index: /usr/src/sys/ia64/ia64/pmap.c =================================================================== --- /usr/src/sys/ia64/ia64/pmap.c (revision 254608) +++ /usr/src/sys/ia64/ia64/pmap.c (working copy) @@ -622,6 +622,8 @@ pmap_free_rid(uint32_t rid) void pmap_pinit0(struct pmap *pmap) { + + PMAP_LOCK_INIT(pmap); /* kernel_pmap is the same as any other pmap. */ pmap_pinit(pmap); } @@ -635,7 +637,6 @@ pmap_pinit(struct pmap *pmap) { int i; - PMAP_LOCK_INIT(pmap); for (i = 0; i < IA64_VM_MINKERN_REGION; i++) pmap->pm_rid[i] = pmap_allocate_rid(); TAILQ_INIT(&pmap->pm_pvchunk); @@ -660,7 +661,6 @@ pmap_release(pmap_t pmap) for (i = 0; i < IA64_VM_MINKERN_REGION; i++) if (pmap->pm_rid[i]) pmap_free_rid(pmap->pm_rid[i]); - PMAP_LOCK_DESTROY(pmap); } /* Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 254608) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -108,7 +108,6 @@ static void vm_hold_load_pages(struct buf *bp, vm_ static void vfs_page_set_valid(struct buf *bp, vm_ooffset_t off, vm_page_t m); static void vfs_page_set_validclean(struct buf *bp, vm_ooffset_t off, vm_page_t m); -static void vfs_drain_busy_pages(struct buf *bp); static void vfs_clean_pages_dirty_buf(struct buf *bp); static void vfs_setdirty_locked_object(struct buf *bp); static void vfs_vmio_release(struct buf *bp); @@ -3983,7 +3982,7 @@ vfs_page_set_validclean(struct buf *bp, vm_ooffset * Ensure that all buffer pages are not exclusive busied. If any page is * exclusive busy, drain it. */ -static void +void vfs_drain_busy_pages(struct buf *bp) { vm_page_t m; Index: /usr/src/sys/kern/vfs_cluster.c =================================================================== --- /usr/src/sys/kern/vfs_cluster.c (revision 254608) +++ /usr/src/sys/kern/vfs_cluster.c (working copy) @@ -315,7 +315,7 @@ cluster_rbuild(struct vnode *vp, u_quad_t filesize daddr_t bn; off_t off; long tinc, tsize; - int i, inc, j, toff; + int i, inc, j, k, toff; KASSERT(size == vp->v_mount->mnt_stat.f_iosize, ("cluster_rbuild: size %ld != f_iosize %jd\n", @@ -378,7 +378,15 @@ cluster_rbuild(struct vnode *vp, u_quad_t filesize inc = btodb(size); bo = &vp->v_bufobj; for (bn = blkno, i = 0; i < run; ++i, bn += inc) { - if (i != 0) { + if (i == 0) { + VM_OBJECT_WLOCK(tbp->b_bufobj->bo_object); + vfs_drain_busy_pages(tbp); + vm_object_pip_add(tbp->b_bufobj->bo_object, + tbp->b_npages); + for (k = 0; k < tbp->b_npages; k++) + vm_page_sbusy(tbp->b_pages[k]); + VM_OBJECT_WUNLOCK(tbp->b_bufobj->bo_object); + } else { if ((bp->b_npages * PAGE_SIZE) + round_page(size) > vp->v_mount->mnt_iosize_max) { break; @@ -424,14 +432,23 @@ cluster_rbuild(struct vnode *vp, u_quad_t filesize if ((tbp->b_pages[j]->valid & vm_page_bits(toff, tinc)) != 0) break; + if (vm_page_xbusied(tbp->b_pages[j])) + break; + vm_object_pip_add(tbp->b_bufobj->bo_object, 1); + vm_page_sbusy(tbp->b_pages[j]); off += tinc; tsize -= tinc; } - VM_OBJECT_WUNLOCK(tbp->b_bufobj->bo_object); if (tsize > 0) { +clean_sbusy: + vm_object_pip_add(tbp->b_bufobj->bo_object, -j); + for (k = 0; k < j; k++) + vm_page_sunbusy(tbp->b_pages[k]); + VM_OBJECT_WUNLOCK(tbp->b_bufobj->bo_object); bqrelse(tbp); break; } + VM_OBJECT_WUNLOCK(tbp->b_bufobj->bo_object); /* * Set a read-ahead mark as appropriate @@ -451,8 +468,8 @@ cluster_rbuild(struct vnode *vp, u_quad_t filesize if (tbp->b_blkno == tbp->b_lblkno) { tbp->b_blkno = bn; } else if (tbp->b_blkno != bn) { - brelse(tbp); - break; + VM_OBJECT_WLOCK(tbp->b_bufobj->bo_object); + goto clean_sbusy; } } /* @@ -466,10 +483,8 @@ cluster_rbuild(struct vnode *vp, u_quad_t filesize for (j = 0; j < tbp->b_npages; j += 1) { vm_page_t m; m = tbp->b_pages[j]; - vm_page_sbusy(m); - vm_object_pip_add(m->object, 1); if ((bp->b_npages == 0) || - (bp->b_pages[bp->b_npages-1] != m)) { + (bp->b_pages[bp->b_npages-1] != m)) { bp->b_pages[bp->b_npages] = m; bp->b_npages++; } @@ -822,7 +837,9 @@ cluster_wbuild(struct vnode *vp, long size, daddr_ (tbp->b_bcount != tbp->b_bufsize) || (tbp->b_bcount != size) || (len == 1) || - ((bp = getpbuf(&cluster_pbuf_freecnt)) == NULL)) { + ((bp = (vp->v_vflag & VV_MD) ? + trypbuf(&cluster_pbuf_freecnt) : + getpbuf(&cluster_pbuf_freecnt)) == NULL)) { totalwritten += tbp->b_bufsize; bawrite(tbp); ++start_lbn; @@ -944,7 +961,9 @@ cluster_wbuild(struct vnode *vp, long size, daddr_ vm_page_t m; VM_OBJECT_WLOCK(tbp->b_bufobj->bo_object); - if (i != 0) { /* if not first buffer */ + if (i == 0) { + vfs_drain_busy_pages(tbp); + } else { /* if not first buffer */ for (j = 0; j < tbp->b_npages; j += 1) { m = tbp->b_pages[j]; if (vm_page_xbusied(m)) { Index: /usr/src/sys/mips/mips/pmap.c =================================================================== --- /usr/src/sys/mips/mips/pmap.c (revision 254608) +++ /usr/src/sys/mips/mips/pmap.c (working copy) @@ -1070,8 +1070,6 @@ pmap_pinit(pmap_t pmap) vm_page_t ptdpg; int i; - PMAP_LOCK_INIT(pmap); - /* * allocate the page directory page */ @@ -1231,7 +1229,6 @@ pmap_release(pmap_t pmap) ptdpg->wire_count--; atomic_subtract_int(&cnt.v_wire_count, 1); vm_page_free_zero(ptdpg); - PMAP_LOCK_DESTROY(pmap); } /* Index: /usr/src/sys/powerpc/aim/mmu_oea.c =================================================================== --- /usr/src/sys/powerpc/aim/mmu_oea.c (revision 254608) +++ /usr/src/sys/powerpc/aim/mmu_oea.c (working copy) @@ -1655,7 +1655,6 @@ moea_pinit(mmu_t mmu, pmap_t pmap) u_int entropy; KASSERT((int)pmap < VM_MIN_KERNEL_ADDRESS, ("moea_pinit: virt pmap")); - PMAP_LOCK_INIT(pmap); RB_INIT(&pmap->pmap_pvo); entropy = 0; @@ -1719,6 +1718,7 @@ void moea_pinit0(mmu_t mmu, pmap_t pm) { + PMAP_LOCK_INIT(pm); moea_pinit(mmu, pm); bzero(&pm->pm_stats, sizeof(pm->pm_stats)); } @@ -1824,7 +1824,6 @@ moea_release(mmu_t mmu, pmap_t pmap) idx /= VSID_NBPW; moea_vsid_bitmap[idx] &= ~mask; mtx_unlock(&moea_vsid_mutex); - PMAP_LOCK_DESTROY(pmap); } /* Index: /usr/src/sys/powerpc/aim/mmu_oea64.c =================================================================== --- /usr/src/sys/powerpc/aim/mmu_oea64.c (revision 254608) +++ /usr/src/sys/powerpc/aim/mmu_oea64.c (working copy) @@ -1879,7 +1879,7 @@ moea64_get_unique_vsid(void) { void moea64_pinit(mmu_t mmu, pmap_t pmap) { - PMAP_LOCK_INIT(pmap); + RB_INIT(&pmap->pmap_pvo); pmap->pm_slb_tree_root = slb_alloc_tree(); @@ -1893,7 +1893,6 @@ moea64_pinit(mmu_t mmu, pmap_t pmap) int i; uint32_t hash; - PMAP_LOCK_INIT(pmap); RB_INIT(&pmap->pmap_pvo); if (pmap_bootstrapped) @@ -1920,6 +1919,8 @@ moea64_pinit(mmu_t mmu, pmap_t pmap) void moea64_pinit0(mmu_t mmu, pmap_t pm) { + + PMAP_LOCK_INIT(pm); moea64_pinit(mmu, pm); bzero(&pm->pm_stats, sizeof(pm->pm_stats)); } @@ -2074,8 +2075,6 @@ moea64_release(mmu_t mmu, pmap_t pmap) moea64_release_vsid(VSID_TO_HASH(pmap->pm_sr[0])); #endif - - PMAP_LOCK_DESTROY(pmap); } /* Index: /usr/src/sys/powerpc/booke/pmap.c =================================================================== --- /usr/src/sys/powerpc/booke/pmap.c (revision 254608) +++ /usr/src/sys/powerpc/booke/pmap.c (working copy) @@ -1477,6 +1477,7 @@ static void mmu_booke_pinit0(mmu_t mmu, pmap_t pmap) { + PMAP_LOCK_INIT(pmap); mmu_booke_pinit(mmu, pmap); PCPU_SET(curpmap, pmap); } @@ -1495,7 +1496,6 @@ mmu_booke_pinit(mmu_t mmu, pmap_t pmap) KASSERT((pmap != kernel_pmap), ("pmap_pinit: initializing kernel_pmap")); - PMAP_LOCK_INIT(pmap); for (i = 0; i < MAXCPU; i++) pmap->pm_tid[i] = TID_NONE; CPU_ZERO(&kernel_pmap->pm_active); @@ -1516,8 +1516,6 @@ mmu_booke_release(mmu_t mmu, pmap_t pmap) KASSERT(pmap->pm_stats.resident_count == 0, ("pmap_release: pmap resident count %ld != 0", pmap->pm_stats.resident_count)); - - PMAP_LOCK_DESTROY(pmap); } /* Index: /usr/src/sys/sparc64/sparc64/pmap.c =================================================================== --- /usr/src/sys/sparc64/sparc64/pmap.c (revision 254608) +++ /usr/src/sys/sparc64/sparc64/pmap.c (working copy) @@ -1204,8 +1204,6 @@ pmap_pinit(pmap_t pm) vm_page_t m; int i; - PMAP_LOCK_INIT(pm); - /* * Allocate KVA space for the TSB. */ @@ -1299,7 +1297,6 @@ pmap_release(pmap_t pm) vm_page_free_zero(m); } VM_OBJECT_WUNLOCK(obj); - PMAP_LOCK_DESTROY(pm); } /* Index: /usr/src/sys/sys/buf.h =================================================================== --- /usr/src/sys/sys/buf.h (revision 254608) +++ /usr/src/sys/sys/buf.h (working copy) @@ -501,6 +501,7 @@ void bufstrategy(struct bufobj *, struct buf *); void brelse(struct buf *); void bqrelse(struct buf *); int vfs_bio_awrite(struct buf *); +void vfs_drain_busy_pages(struct buf *bp); struct buf * getpbuf(int *); struct buf *incore(struct bufobj *, daddr_t); struct buf *gbincore(struct bufobj *, daddr_t); Index: /usr/src/sys/vm/vm_map.c =================================================================== --- /usr/src/sys/vm/vm_map.c (revision 254608) +++ /usr/src/sys/vm/vm_map.c (working copy) @@ -226,6 +226,7 @@ vmspace_zinit(void *mem, int size, int flags) vm->vm_map.pmap = NULL; (void)vm_map_zinit(&vm->vm_map, sizeof(vm->vm_map), flags); + PMAP_LOCK_INIT(vmspace_pmap(vm)); return (0); } Index: /usr/src/sys/vm/vm_page.c =================================================================== --- /usr/src/sys/vm/vm_page.c (revision 254608) +++ /usr/src/sys/vm/vm_page.c (working copy) @@ -2558,6 +2558,8 @@ vm_page_cache(vm_page_t m) vm_page_free(m); return; } + if (!cache_was_empty) + cache_was_empty = vm_radix_is_singleton(&object->cache); m->flags |= PG_CACHED; cnt.v_cache_count++; PCPU_INC(cnt.v_tcached); Index: /usr/src/sys/vm/vm_radix.c =================================================================== --- /usr/src/sys/vm/vm_radix.c (revision 254608) +++ /usr/src/sys/vm/vm_radix.c (working copy) @@ -432,6 +432,20 @@ restart: } /* + * XXX + */ +boolean_t +vm_radix_is_singleton(struct vm_radix *rtree) +{ + struct vm_radix_node *rnode; + + rnode = vm_radix_getroot(rtree); + if (rnode == NULL) + return (FALSE); + return (vm_radix_isleaf(rnode)); +} + +/* * Returns the value stored at the index. If the index is not present, * NULL is returned. */ Index: /usr/src/sys/vm/vm_radix.h =================================================================== --- /usr/src/sys/vm/vm_radix.h (revision 254608) +++ /usr/src/sys/vm/vm_radix.h (working copy) @@ -37,6 +37,7 @@ void vm_radix_init(void); int vm_radix_insert(struct vm_radix *rtree, vm_page_t page); +boolean_t vm_radix_is_singleton(struct vm_radix *rtree); vm_page_t vm_radix_lookup(struct vm_radix *rtree, vm_pindex_t index); vm_page_t vm_radix_lookup_ge(struct vm_radix *rtree, vm_pindex_t index); vm_page_t vm_radix_lookup_le(struct vm_radix *rtree, vm_pindex_t index); -- Test scenario: suj9.sh