GENERIC HEAD from 2012-05-17 10:19:49 UTC, r235546M, vmcore.355 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r235546M: Thu May 17 17:06:47 CEST 2012 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531575296 (3367 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 621141 free (1357 frags, 77473 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 123156 free (5316 frags, 14730 blocks, 0.5% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 80775251 free (443 frags, 10096851 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 3321787 free (234107 frags, 385960 blocks, 2.3% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 11194878 free (80422 frags, 1389307 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN re0: link state changed to UP Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa inet 127.0.0.1 netmask 0xff000000 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (100baseTX ) status: active fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/gcc46 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Starting nfsuserd. Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xcaf63760 ufs (ufs) @ kern/vfs_lookup.c:542 2nd 0xe1928ac0 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 3rd 0xcaf05d00 ufs (ufs) @ kern/vfs_subr.c:2160 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,3a632e73,a303632,ffffff00,c119d440,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6fd,c12d92b8,870,f272b33c,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6fd,caf05d00,c0fc3a04,c7d89610,c0fe375a,...) at _witness_debugger+0x25 witness_checkorder(caf05d00,9,c0fe3751,870,0,...) at witness_checkorder+0x86f __lockmgr_args(caf05d00,80100,caf05d6c,0,0,...) at __lockmgr_args+0x8c5 ffs_lock(f272b444,c0ad5b8c,c0fe0b8b,5e2,c1109e88,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c1139660,f272b444,ca8663a4,c114b840,caf05ca8,...) at VOP_LOCK1_APV+0xf3 _vn_lock(caf05ca8,80100,c0fe3751,870,eb,...) at _vn_lock+0x78 vget(caf05ca8,80100,ca8662f0,50,0,...) at vget+0xbb vfs_hash_get(ca780b70,15ef55,80000,ca8662f0,f272b590,...) at vfs_hash_get+0xed ffs_vgetf(ca780b70,15ef55,80000,f272b590,1,...) at ffs_vgetf+0x49 softdep_sync_buf(caf63708,e1928a60,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(caf63708,1,0,ca8663a4,c12d92a8,...) at ffs_syncvnode+0x24c ffs_truncate(caf63708,200,0,880,ca803280,...) at ffs_truncate+0x8b5 ufs_direnter(caf63708,caf05ca8,f272b904,f272bbb8,0,...) at ufs_direnter+0x980 ufs_makeinode(f272bbb8,c1139c20,f272bafc,f272ba58,c0e29c90,...) at ufs_makeinode+0x65d ufs_create(f272bafc,c102e96e,c114bac0,caf63708,f272bb14,...) at ufs_create+0x30 VOP_CREATE_APV(c1139660,f272bafc,f272bbb8,f272ba94,0,...) at VOP_CREATE_APV+0x100 vn_open_cred(f272bb78,f272bc40,1b0,0,ca803280,...) at vn_open_cred+0x205 vn_open(f272bb78,f272bc40,1b0,ca7aca48,4fb51579,...) at vn_open+0x3b kern_openat(ca8662f0,ffffff9c,886cf60,0,a03,...) at kern_openat+0x1ec kern_open(ca8662f0,886cf60,0,a02,1b0,...) at kern_open+0x35 sys_open(ca8662f0,f272bcec,c102715c,c0fdc3ef,c1028770,...) at sys_open+0x30 syscall(f272bd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283d01c3, esp = 0xbfbfc4ec, ebp = 0xbfbfc578 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xcab1e8c8 ufs (ufs) @ kern/vfs_mount.c:1254 2nd 0xca87dd00 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1834 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,38313a63,a3433,cadae328,8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12d8ee0,72a,f274f9a8,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,ca87dd00,c0fce934,c7d89470,c1009255,...) at _witness_debugger+0x25 witness_checkorder(ca87dd00,9,c100924c,72a,ca87dd6c,...) at witness_checkorder+0x86f __lockmgr_args(ca87dd00,80400,ca87dd6c,0,0,...) at __lockmgr_args+0x8c5 vop_stdlock(f274faac,c130e680,4,c0fd1a67,eb,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10f9240,f274faac,c130e680,c114b840,ca87dca8,...) at VOP_LOCK1_APV+0xf3 _vn_lock(ca87dca8,80400,c100924c,72a,0,...) at _vn_lock+0x78 softdep_flushworklist(cab202dc,f274fb64,ca7e1000,612,ca87d0c4,...) at softdep_flushworklist+0x47 ffs_sync(cab202dc,1,f274fbb8,51f,c1baf630,...) at ffs_sync+0x3bf dounmount(cab202dc,8080000,ca7e1000,4a1,f274fc3c,...) at dounmount+0x3f8 sys_unmount(ca7e1000,f274fcec,c102715c,c0fdc3b2,c1028770,...) at sys_unmount+0x421 syscall(f274fd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280bdd6b, esp = 0xbfbfe63c, ebp = 0xbfbfe708 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 19718 files, 37538730 used, 80775251 free (443 frags, 10096851 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 19718 files, 37538730 used, 80775251 free (443 frags, 10096851 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Thu May 17 17:18:08 CEST 2012 FreeBSD/i386 (x4.osted.lan) (console) login: Expensive timeout(9) function: 0xc08ce250(0xc132fc60) 0.012870215 s lock order reversal: 1st 0xe19393a0 bufwait (bufwait) @ kern/vfs_bio.c:2652 2nd 0xca89fa00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,2e687361,38323a63,c1000a34,c96253a4,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12d9060,11c,f27a37c8,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,ca89fa00,c100d15a,c7d89678,c100cdec,...) at _witness_debugger+0x25 witness_checkorder(ca89fa00,9,c100cde3,11c,0,...) at witness_checkorder+0x86f _sx_xlock(ca89fa00,0,c100cde3,11c,cada0ae0,...) at _sx_xlock+0x9a ufsdirhash_acquire(e1939340,cada0ae0,104,e281a314,f27a3898,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(cada0ae0,f27a3904,2314,f27a3884,f27a3888,...) at ufsdirhash_add+0x13 ufs_direnter(caf7c870,cb1a4870,f27a3904,f27a3bb8,0,...) at ufs_direnter+0x799 ufs_makeinode(f27a3bb8,c1139c20,f27a3afc,f27a3a58,c0e29c90,...) at ufs_makeinode+0x65d ufs_create(f27a3afc,c102e96e,c114bac0,caf7c870,f27a3b14,...) at ufs_create+0x30 VOP_CREATE_APV(c1139660,f27a3afc,f27a3bb8,f27a3a94,0,...) at VOP_CREATE_APV+0x100 vn_open_cred(f27a3b78,f27a3c40,180,0,ca803000,...) at vn_open_cred+0x205 vn_open(f27a3b78,f27a3c40,180,ca7f1658,0,...) at vn_open+0x3b kern_openat(c96252f0,ffffff9c,bfbfe7ab,0,a02,...) at kern_openat+0x1ec kern_open(c96252f0,bfbfe7ab,0,a01,180,...) at kern_open+0x35 sys_open(c96252f0,f27a3cec,c102715c,c0fdc3ef,c0ab1df7,...) at sys_open+0x30 syscall(f27a3d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2829d1c3, esp = 0xbfbfe37c, ebp = 0xbfbfec18 --- May 17 17:31:54 x4 su: BAD SU pho to root on /dev/pts/0 May 17 17:31:58 x4 su: pho to root on /dev/pts/0 20120517 17:32:06 all: datamove.sh 20120517 17:32:08 all: datamove2.sh 20120517 18:05:04 all: datamove3.sh 20120517 18:05:05 all: datamove.sh 20120517 18:05:06 all: datamove2.sh 20120517 18:19:41 all: datamove.sh 20120517 18:19:42 all: datamove2.sh Expensive timeout(9) function: 0xc0c35f70(0xcb14d810) 0.488445411 s 20120517 18:52:14 all: datamove3.sh 20120517 18:52:15 all: lor2.sh 20120517 18:52:48 all: datamove.sh 20120517 18:52:56 all: datamove2.sh 20120517 19:25:50 all: datamove3.sh 20120517 19:25:51 all: lor2.sh 20120517 19:26:25 all: datamove.sh 20120517 19:26:32 all: datamove2.sh 20120517 19:58:45 all: datamove3.sh 20120517 19:58:46 all: lor2.sh 20120517 19:59:34 all: datamove.sh 20120517 19:59:40 all: datamove2.sh 20120517 20:32:10 all: revoke.sh 20120517 20:32:11 all: kevent6.sh 20120517 20:32:39 all: nfs8.sh May 17 20:32:39 x4 mountd[1009]: bad exports list line /mnt 20120517 22:01:37 all: trim2.sh 20120517 22:01:51 all: kevent9.sh 20120517 22:04:03 all: mmap.sh 20120517 22:04:04 all: md2.sh lock order reversal: 1st 0xcbcd221c ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xcd422a30 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1690 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,66000a30,66752f73,78655f73,74746174,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12d5ed0,69a,f30df748,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,cd422a30,c0fc3a04,c7d89610,c100c139,...) at _witness_debugger+0x25 witness_checkorder(cd422a30,9,c100c130,69a,0,...) at witness_checkorder+0x86f __lockmgr_args(cd422a30,80000,0,0,0,...) at __lockmgr_args+0x8c5 ffs_vgetf(ca77f5b8,2,80000,f30df87c,0,...) at ffs_vgetf+0x189 ffs_vget(ca77f5b8,2,80000,f30df87c,0,...) at ffs_vget+0x2e ufs_root(ca77f5b8,80000,f30df8a4,cbcd2200,c0fd1a67,...) at ufs_root+0x28 ufs_extattr_autostart(ca77f5b8,cb6abbc0,c100c130,42c,0,...) at ufs_extattr_autostart+0x4d ffs_mount(ca77f5b8,c1139220,cbd0a480,ca7cc900,965,...) at ffs_mount+0x2e77 vfs_donmount(cb6abbc0,0,0,cdabcd80,cdabcd80,...) at vfs_donmount+0xc0a sys_nmount(cb6abbc0,f30dfcec,c102715c,c0fdc3ef,c0ab1df7,...) at sys_nmount+0x7f syscall(f30dfd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280cf98b, esp = 0xbfbfd84c, ebp = 0xbfbfdda8 --- Expensive timeout(9) function: 0xc0cfd8e0(0) 0.903800008 s 20120517 22:25:57 all: suj19.sh May 17 22:25:59 x4 kernel: pid 72478 (rw), uid 1004 inumber 32777 on /mnt: filesystem full May 17 22:28:00 x4 kernel: pid 72708 (rw), uid 1004 inumber 98311 on /mnt: filesystem full May 17 22:29:59 x4 kernel: pid 72955 (rw), uid 1004 inumber 32786 on /mnt: filesystem full May 17 22:36:00 x4 kernel: pid 73645 (rw), uid 1004 inumber 32776 on /mnt: filesystem full May 17 22:38:00 x4 kernel: pid 73855 (rw), uid 1004 inumber 98323 on /mnt: filesystem full May 17 22:39:59 x4 kernel: pid 74104 (rw), uid 1004 inumber 98349 on /mnt: filesystem full May 17 22:42:01 x4 kernel: pid 74335 (rw), uid 1004 inumber 32782 on /mnt: filesystem full 20120517 22:46:01 all: rename4.sh 20120517 22:46:02 all: msdos3.sh 20120517 22:46:03 all: fpclone2.sh 20120517 22:46:07 all: pageout.sh 20120517 22:59:40 all: crossmp2.sh 20120517 23:00:42 all: sendfile.sh 20120517 23:07:08 all: jail3.sh 20120517 23:07:09 all: lor2.sh 20120517 23:07:41 all: mac.sh 20120517 23:07:48 all: sysctl.sh 20120517 23:07:58 all: recursiveflushes.sh lock order reversal: 1st 0xe1a52400 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 2nd 0xcef4be1c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2297 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,632e746f,3932323a,f2000a37,f2ce22c0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12ca480,8f9,f2ce234c,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,cef4be1c,c1008d14,c7d8a170,c1008c9e,...) at _witness_debugger+0x25 witness_checkorder(cef4be1c,9,c1008c95,8f9,0,...) at witness_checkorder+0x86f __lockmgr_args(cef4be1c,80900,ca43922c,0,0,...) at __lockmgr_args+0x8c5 ffs_copyonwrite(ca439168,e1a523a0,0,e1a523a0,cfc6c870,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(ca439260,e1a523a0,c12ca494,c7d87ba8,cfc6c870,...) at ffs_geom_strategy+0xeb ufs_strategy(f2ce24f0,c102e66b,4,c0fd1a67,eb,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c1139660,f2ce24f0,cfc6c968,c114ba20,cfc6c870,...) at VOP_STRATEGY_APV+0x119 bufstrategy(cfc6c968,e1a523a0,e1b06ce0,e1a52400,e1a523a0,...) at bufstrategy+0x61 bufwrite(e1a523a0,e1a523a0,1,106,0,...) at bufwrite+0x159 ffs_syncvnode(cfc6c870,1,0,4000,c7d61b80,...) at ffs_syncvnode+0x385 ffs_snapshot(ca780b70,cb62d4c0,c100c130,1a9,0,...) at ffs_snapshot+0x99c ffs_mount(ca780b70,cd624a00,ff,3b8,f2ce2a80,...) at ffs_mount+0x1c92 vfs_donmount(cb529000,211000,0,cde9e000,cde9e000,...) at vfs_donmount+0x1354 sys_nmount(cb529000,f2ce2cec,c102715c,c0fbde51,c0ab1df7,...) at sys_nmount+0x7f syscall(f2ce2d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280bb98b, esp = 0xbfbfe2ac, ebp = 0xbfbfea28 --- lock order reversal: 1st 0xcef4be1c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:966 2nd 0xcf5a8058 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1627 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,a373236,a00,c7ee8bc0,f2aca694,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12cb288,65b,f2aca718,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,cf5a8058,c0fc3a04,c7d89610,c1008c9e,...) at _witness_debugger+0x25 witness_checkorder(cf5a8058,9,c1008c95,65b,0,...) at witness_checkorder+0x86f __lockmgr_args(cf5a8058,80000,0,0,0,...) at __lockmgr_args+0x8c5 ffs_snapremove(cf5a8000,c1008bd3,3,1467c1f,cb4d2000,...) at ffs_snapremove+0x11f ffs_truncate(cf5a8000,0,0,c00,0,...) at ffs_truncate+0x647 ufs_inactive(f2acaad4,c102e33d,eb,c1139660,f2acaaa8,...) at ufs_inactive+0x1f6 VOP_INACTIVE_APV(c1139660,f2acaad4,c0fe3751,9c1,f2acab08,...) at VOP_INACTIVE_APV+0x100 vinactive(cf5a8000,cb4d2000,c0fe3751,8bf,0,...) at vinactive+0x127 vputx(f2acac34,c0b2fd53,cf5a8000,ffffffdf,cf5a8000,...) at vputx+0x370 vput(cf5a8000,ffffffdf,cf5a8000,f2acab8c,1bb50,...) at vput+0x10 kern_unlinkat(cb4d2000,ffffff9c,bfbfebda,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(cb4d2000,bfbfebda,0,f2acad1c,c0e01a2e,...) at kern_unlink+0x2f sys_unlink(cb4d2000,f2acacec,c102715c,c0fdc403,c1028770,...) at sys_unlink+0x22 syscall(f2acad28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x28160e1f, esp = 0xbfbfe94c, ebp = 0xbfbfe9c8 --- 20120517 23:12:00 all: mount.sh 20120517 23:12:01 all: lookup_shared2.sh 20120517 23:22:47 all: core.sh 20120517 23:26:01 all: mountro2.sh 20120517 23:26:03 all: mmap4.sh vnode_pager_putpages: I/O error 28 vnode_pager_putpages: residual I/O 32768 at 9776 May 17 23:26:03 x4 kernel: pid 96381 (mmap4), uid 0 inumber 4 on /mnt: filesystem full 20120517 23:26:05 all: mountro3.sh 20120517 23:26:07 all: softupdate.sh 20120518 00:07:33 all: rename6.sh 20120518 00:07:39 all: suj9.sh 20120518 01:05:23 all: selfd.sh May 18 01:05:24 x4 mountd[1009]: bad exports list line /mnt May 18 01:05:55 x4 last message repeated 51 times 20120518 01:06:24 all: quota1.sh May 18 01:06:23 x4 last message repeated 47 times 20120518 01:16:35 all: suj2.sh 20120518 01:16:36 all: trim5.sh 20120518 01:16:36 all: sndstat.sh 20120518 01:16:37 all: suj14.sh May 18 01:18:17 x4 kernel: pid 14677 (cp), uid 0 inumber 52101 on /mnt: filesystem full May 18 01:18:18 x4 kernel: pid 14679 (cp), uid 0 inumber 52118 on /mnt: filesystem full May 18 01:18:19 x4 kernel: pid 14695 (cp), uid 0 inumber 52118 on /mnt: filesystem full 20120518 01:18:23 all: kinfo.sh 20120518 01:48:24 all: malloc2.sh 20120518 01:50:45 all: nfs5.sh 20120518 01:51:48 all: kinfo3.sh 20120518 02:07:19 all: suj6.sh 20120518 02:08:01 all: tmpfs4.sh 20120518 02:08:02 all: syscall5.sh 20120518 02:38:11 all: nfsrename.sh May 18 02:38:11 x4 mountd[1009]: bad exports list line /mnt 20120518 02:41:02 all: linger4.sh 20120518 03:18:22 all: kevent.sh 20120518 03:19:31 all: lag.sh 20120518 03:30:38 all: kevent2.sh 20120518 03:30:46 all: msdos4.sh lock order reversal: 1st 0xcaa62e20 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:148 2nd 0xe18e9320 bufwait (bufwait) @ vm/vm_pager.c:310 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,75000a30,3a632e66,a383431,c133d200,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12d46b0,136,f31155ec,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,e18e9320,c0fe16cf,c7d85370,c1011dbf,...) at _witness_debugger+0x25 witness_checkorder(e18e9320,9,c1011db6,136,0,...) at witness_checkorder+0x86f __lockmgr_args(e18e9320,80000,0,0,0,...) at __lockmgr_args+0x8c5 initpbuf(c13101ec,0,c1011db6,161,0,...) at initpbuf+0xe0 getpbuf(c130e850,0,c100e274,47f,0,...) at getpbuf+0x12f swap_pager_getpages(d1d039d4,f3115750,3,0,f31158cc,...) at swap_pager_getpages+0x1bd vm_fault_hold(cb6230e8,8846000,1,0,0,...) at vm_fault_hold+0xe63 vm_fault(cb6230e8,8846000,1,0,88460f8,...) at vm_fault+0x8b trap_pfault(1f4,0,23d61c6,23d8b7f,cb67c2cc,...) at trap_pfault+0x1b9 trap(f3115a60) at trap+0x5f1 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0dff312, esp = 0xf3115aa0, ebp = 0xf3115adc --- copyin(0,f3115b1c,c0af2d86,cabf1218,4,...) at copyin+0x32 uiomove(cabf1218,4,f3115c3c,1,0,...) at uiomove+0x1b m_uiotombuf(f3115c3c,2,5,0,0,...) at m_uiotombuf+0xf6 sosend_generic(caa62d00,0,f3115c3c,0,0,...) at sosend_generic+0x432 sosend(caa62d00,0,f3115c3c,0,0,...) at sosend+0x3f soo_write(ca7f1620,f3115c3c,ca7cc900,0,cb69f2f0,...) at soo_write+0x63 dofilewrite(f3115c3c,ffffffff,ffffffff,0,f3115c18,...) at dofilewrite+0x95 kern_writev(cb69f2f0,3,f3115c3c,f3115c5c,1,...) at kern_writev+0x68 sys_write(cb69f2f0,f3115cec,c102715c,c0fa6ae2,dddd358b,...) at sys_write+0x4f syscall(f3115d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281857a3, esp = 0xbfbfe74c, ebp = 0xbfbfe7a8 --- 20120518 03:52:07 all: rename8.sh 20120518 03:52:58 all: gjournal.sh GEOM_JOURNAL: Journal 603561551: md5 contains data. GEOM_JOURNAL: Journal 603561551: md5 contains journal. GEOM_JOURNAL: Journal md5 clean. GEOM_JOURNAL: BIO_FLUSH not supported by md5. May 18 03:53:14 x4 kernel: pid 14924 (dd), uid 0 inumber 4 on /media: filesystem full 20120518 03:53:16 all: fpclone.sh 20120518 03:55:20 all: union.sh 20120518 04:06:45 all: crossmp.sh fsync: giving up on dirty 0xd32d29d8: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xcbc30800 flags (VI(0x200)) v_object 0xd0a63378 ref 0 pages 10 lock type devfs: EXCL by thread 0xcb4d22f0 (pid 17974, umount, tid 100313) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cdedb7 at ffs_sync+0x467 #5 0xc0b20dd8 at dounmount+0x3f8 #6 0xc0b214c1 at sys_unmount+0x421 #7 0xc0e01a2e at syscall+0x2fe #8 0xc0debe91 at Xint0x80_syscall+0x21 dev md14a May 18 04:06:53 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:06:55 x4 mountd[1009]: can't delete exports for /mnt5: Device busy May 18 04:06:55 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:06:55 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument fsync: giving up on dirty 0xcf393438: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xcb899c00 flags (VI(0x200)) v_object 0xce87bd4c ref 0 pages 10 lock type devfs: EXCL by thread 0xcabf92f0 (pid 18468, umount, tid 100667) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cdedb7 at ffs_sync+0x467 #5 0xc0b20dd8 at dounmount+0x3f8 #6 0xc0b214c1 at sys_unmount+0x421 #7 0xc0e01a2e at syscall+0x2fe #8 0xc0debe91 at Xint0x80_syscall+0x21 dev md5a May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:07:02 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:07:03 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument : : May 18 04:08:42 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:08:42 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:08:42 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:08:42 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument fsync: giving up on dirty 0xd0eb7b40: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xcba84500 flags (VI(0x200)) v_object 0xccb31de0 ref 0 pages 10 lock type devfs: EXCL by thread 0xd4afd000 (pid 33651, umount, tid 3114205) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cdedb7 at ffs_sync+0x467 #5 0xc0b20dd8 at dounmount+0x3f8 #6 0xc0b214c1 at sys_unmount+0x421 #7 0xc0e01a2e at syscall+0x2fe #8 0xc0debe91 at Xint0x80_syscall+0x21 dev md17a May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:08:47 x4 mountd[1009]: can't delete exports for /mnt12: Device busy May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 04:08:51 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:08:53 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:08:53 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:08:53 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:08:53 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:08:53 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:08:54 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 04:08:54 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:08:55 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:08:55 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:08:55 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:08:55 x4 mountd[1009]: can't delete exports for /mnt9: Device busy May 18 04:08:59 x4 mountd[1009]: can't delete exports for /mnt8: Device busy May 18 04:09:03 x4 mountd[1009]: can't delete exports for /mnt6: Device busy May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 04:09:04 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:09:05 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:09:05 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:09:06 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:09:09 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:09:09 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:09:09 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:09:09 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:09:16 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt16: Invalid argument May 18 04:09:18 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt13: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:09:25 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument May 18 04:09:26 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 04:09:26 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:09:27 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument fsync: giving up on dirty 0xd2576e10: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xcb9f6500 flags (VI(0x200)) v_object 0xd0b8a250 ref 0 pages 10 lock type devfs: EXCL by thread 0xcb5ac5e0 (pid 39348, umount, tid 100382) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cdedb7 at ffs_sync+0x467 #5 0xc0b20dd8 at dounmount+0x3f8 #6 0xc0b214c1 at sys_unmount+0x421 #7 0xc0e01a2e at syscall+0x2fe #8 0xc0debe91 at Xint0x80_syscall+0x21 dev md10a May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt7: Device busy May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt18: Invalid argument May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt14: Invalid argument May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt17: Invalid argument May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt11: Invalid argument May 18 04:09:28 x4 mountd[1009]: can't delete exports for /mnt15: Invalid argument : : May 18 04:26:16 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 04:26:16 x4 mountd[1009]: can't delete exports for /mnt19: Invalid argument May 18 04:26:16 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument 20120518 04:26:20 all: suj.sh 20120518 04:46:30 all: umount.sh 20120518 04:50:39 all: fragments.sh May 18 04:50:49 x4 kernel: pid 61694 (fragments), uid 1004 inumber 85866 on /mnt: filesystem full May 18 04:50:49 x4 kernel: pid 61694 (fragments), uid 1004 inumber 65556 on /mnt: filesystem full May 18 04:50:50 x4 kernel: pid 61718 (fragments), uid 1004 inumber 5146 on /mnt: filesystem full May 18 04:50:51 x4 kernel: pid 61714 (fragments), uid 1004 inumber 5172 on /mnt: filesystem full May 18 04:50:52 x4 kernel: pid 61716 (fragments), uid 1004 inumber 5175 on /mnt: filesystem full May 18 04:50:53 x4 kernel: pid 61719 (fragments), uid 1004 inumber 5177 on /mnt: filesystem full May 18 04:50:54 x4 kernel: pid 61717 (fragments), uid 1004 inumber 5178 on /mnt: filesystem full : : May 18 04:59:57 x4 kernel: pid 62575 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 04:59:58 x4 kernel: pid 62575 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 04:59:59 x4 kernel: pid 62578 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:00 x4 kernel: pid 62575 (fragments), uid 1004 inumber 2 on /mnt: filesystem full lock order reversal: 1st 0xcb544b2c filedesc structure (filedesc structure) @ kern/kern_descrip.c:822 2nd 0xcadaf490 ufs (ufs) @ kern/vfs_subr.c:4317 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,20296572,656b2040,6b2f6e72,5f6e7265,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6e4,c12d4028,10dd,f28f3a68,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6e4,cadaf490,c0fc3a04,c7d89610,c0fe375a,...) at _witness_debugger+0x25 witness_checkorder(cadaf490,9,c0fe3751,10dd,cadaf4fc,...) at witness_checkorder+0x86f __lockmgr_args(cadaf490,80400,cadaf4fc,0,0,...) at __lockmgr_args+0x8c5 ffs_lock(f28f3b70,1,c7d823e8,f28f3b54,c0b15f0a,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c1139660,f28f3b70,c0ad5b8c,c114b840,cadaf438,...) at VOP_LOCK1_APV+0xf3 _vn_lock(cadaf438,80400,c0fe3751,10dd,f28f3bcc,...) at _vn_lock+0x78 vfs_knllock(cadaf438,0,c0fcd385,728,caf70dc8,...) at vfs_knllock+0x29 knlist_remove_kq(0,f28f3bec,c0b253f9,cb5a501c,caf70dc8,...) at knlist_remove_kq+0x85 May 18 05:00:01 x4 kernel: pid 6kn2576 (fragments)lis, uid 1004 inumbt_rer 2 on /mnt: fiemolesystem full ve(cb5a501c,caf70dc8,0,f28f3c18,c0a4fc55,...) at knlist_remove+0x1b filt_vfsdetach(caf70dc8,0,c0fcd385,832,0,...) at filt_vfsdetach+0x39 knote_fdclose(cb1a22f0,3,c0fccacc,336,5,...) at knote_fdclose+0xf5 do_dup(3,cb1a253c,f28f3d1c,c0e01a2e,cb1a22f0,...) at do_dup+0x351 sys_dup2(cb1a22f0,f28f3cec,c102715c,c0fbd045,c0ab1df7,...) at sys_dup2+0x28 syscall(f28f3d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (90, FreeBSD ELF32, sys_dup2), eip = 0x280f6caf, esp = 0xbfbfeabc, ebp = 0xbfbfeb08 --- May 18 05:00:02 x4 kernel: pid 62572 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:03 x4 kernel: pid 62572 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:04 x4 kernel: pid 62579 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:05 x4 kernel: pid 62579 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:06 x4 kernel: pid 62572 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:07 x4 kernel: pid 62574 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 18 05:00:08 x4 kernel: pid 62579 (fragments), uid 1004 inumber 2 on /mnt: filesystem full : : May 18 05:53:46 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5104 on /mnt: filesystem full May 18 05:53:47 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5150 on /mnt: filesystem full May 18 05:53:48 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5203 on /mnt: filesystem full May 18 05:53:49 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5252 on /mnt: filesystem full May 18 05:53:50 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5128 on /mnt: filesystem full May 18 05:53:51 x4 kernel: pid 67674 (fragments), uid 1004 inumber 5278 on /mnt: filesystem full 20120518 05:53:53 all: umountf.sh 20120518 05:55:21 all: sendfile3.sh 20120518 05:56:42 all: nfs7.sh May 18 05:56:43 x4 mountd[1009]: bad exports list line /mnt May 18 06:12:21 x4 kernel: pid 68947 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:12:25 x4 kernel: pid 68950 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:12:43 x4 kernel: pid 68942 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:13:03 x4 kernel: pid 68946 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:13:44 x4 kernel: pid 68949 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:14:31 x4 kernel: pid 68948 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:15:09 x4 kernel: pid 69846 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:02 x4 kernel: pid 68938 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:35 x4 kernel: pid 68943 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:51 x4 kernel: pid 68945 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:51 x4 kernel: pid 69851 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:54 x4 kernel: pid 69848 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:16:59 x4 kernel: pid 71237 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:18:15 x4 kernel: pid 71241 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:18:17 x4 kernel: pid 71243 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:18:17 x4 kernel: pid 71238 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:18:18 x4 kernel: pid 71240 (lockf), uid 0, was killed: exceeded maximum CPU limit May 18 06:18:18 x4 kernel: pid 71239 (lockf), uid 0, was killed: exceeded maximum CPU limit 20120518 06:18:34 all: truncate5.sh 20120518 06:18:38 all: nullfs2.sh 20120518 06:28:59 all: nullfs8.sh lock order reversal: 1st 0xcd5048c8 newnfs (newnfs) @ kern/vfs_syscalls.c:4185 2nd 0xe1bb66e0 bufwait (bufwait) @ kern/vfs_bio.c:1905 3rd 0xcecc28c8 newnfs (newnfs) @ kern/vfs_subr.c:2160 KDB: stack backtrace: db_trace_self_wrapper(c0fd7c17,a3036,a3500,f29643ca,4,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac348b,c0fdb6fd,c12c9e50,870,f2964414,...) at kdb_backtrace+0x2a _witness_debugger(c0fdb6fd,cecc28c8,c0fbe04b,c7d89e30,c0fe375a,...) at _witness_debugger+0x25 witness_checkorder(cecc28c8,9,c0fe3751,870,0,...) at witness_checkorder+0x86f __lockmgr_args(cecc28c8,80100,cecc2934,0,0,...) at __lockmgr_args+0x8c5 nfs_lock1(f2964518,c0ad5b8c,246,c12c9e54,c7d827f8,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c10fa360,f2964518,cc64c3a4,c114b840,cecc2870,...) at VOP_LOCK1_APV+0xf3 _vn_lock(cecc2870,80100,c0fe3751,870,eb,...) at _vn_lock+0x78 vget(cecc2870,80100,cc64c2f0,50,0,...) at vget+0xbb vfs_hash_get(d00e22dc,5dc5634c,80000,cc64c2f0,f29645e0,...) at vfs_hash_get+0xed nfscl_nget(d00e22dc,cd504870,cfa17980,f2964794,cc64c2f0,...) at nfscl_nget+0x8d nfsrpc_readdirplus(cd504870,f2964980,f296493c,cad0ba00,cc64c2f0,...) at nfsrpc_readdirplus+0x13ce ncl_readdirplusrpc(cd504870,f2964980,cad0ba00,cc64c2f0,cb148128,...) at ncl_readdirplusrpc+0xec ncl_doio(cd504870,e1bb6680,cad0ba00,cc64c2f0,0,...) at ncl_doio+0x28d ncl_bioread(cd504870,f2964be4,0,cad0ba00,c0e2331a,...) at ncl_bioread+0xc4d nfs_readdir(f2964c04,c102dde1,200400,c0fe4e27,1059,...) at nfs_readdir+0x192 VOP_READDIR_APV(c10fa360,f2964c04,c0fe4e27,1059,f2964c2c,...) at VOP_READDIR_APV+0x100 kern_getdirentries(cc64c2f0,5,8893000,1000,f2964c58,...) at kern_getdirentries+0x230 sys_getdirentries(cc64c2f0,f2964cec,c102715c,c0fdc3a3,c0ab1df7,...) at sys_getdirentries+0x31 syscall(f2964d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (196, FreeBSD ELF32, sys_getdirentries), eip = 0x280dcedf, esp = 0xbfbfe72c, ebp = 0xbfbfe758 --- 20120518 06:31:18 all: kevent6.sh 20120518 06:31:46 all: suj3.sh cryptosoft0: on motherboard GEOM_ELI: Device md5.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software GEOM_ELI: md5 has been killed. GEOM_ELI: Device md5.eli destroyed. 20120518 06:38:12 all: linger2.sh May 18 06:38:16 x4 kernel: pid 76918 (linger2), uid 1004 inumber 892 on /mnt: filesystem full May 18 06:38:16 x4 kernel: pid 76919 (linger2), uid 1004 inumber 890 on /mnt: filesystem full 20120518 06:40:00 all: snap2.sh 20120518 06:44:00 all: trim.sh 20120518 07:18:02 all: suj13.sh Expensive timeout(9) function: 0xc0cfd8e0(0) 0.915050838 s 20120518 07:38:16 all: suj21.sh 20120518 07:39:42 all: tmpfs2.sh May 18 07:39:43 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:39:43 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:39:44 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:39:44 x4 mountd[1009]: can't delete exports for /mnt12: Invalid argument : : May 18 07:44:13 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:44:13 x4 mountd[1009]: can't delete exports for /mnt12: Invalid argument May 18 07:44:14 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument 20120518 07:44:14 all: newfs4.sh 20120518 07:48:44 all: ldt2.sh 20120518 07:48:45 all: devfs.sh May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:48:45 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:48:46 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:48:46 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:46 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:48:46 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument May 18 07:48:46 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 07:48:47 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:48 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 07:48:48 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:48:49 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 07:48:49 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt8: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:48:50 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:51 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt5: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument May 18 07:48:52 x4 mountd[1009]: can't delete exports for /mnt9: Invalid argument May 18 07:48:53 x4 mountd[1009]: can't delete exports for /mnt10: Invalid argument May 18 07:48:53 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:53 x4 mountd[1009]: can't delete exports for /mnt7: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt3: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt2: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt4: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt6: Invalid argument May 18 07:48:54 x4 mountd[1009]: can't delete exports for /mnt1: Invalid argument 20120518 07:48:55 all: mountro.sh fsync: giving up on dirty 0xca7d5168: tag devfs, type VCHR usecount 1, writecount 0, refcount 78 mountedhere 0xcba84b00 flags (VI(0x200)) v_object 0xd30a84a0 ref 0 pages 1070 lock type devfs: EXCL by thread 0xcb2e58d0 (pid 64002, umount, tid 100175) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cd9d07 at softdep_flushworklist+0x47 #5 0xc0cded0f at ffs_sync+0x3bf #6 0xc0b20dd8 at dounmount+0x3f8 #7 0xc0b214c1 at sys_unmount+0x421 #8 0xc0e01a2e at syscall+0x2fe #9 0xc0debe91 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xca7d5168: tag devfs, type VCHR usecount 1, writecount 0, refcount 78 mountedhere 0xcba84b00 flags (VI(0x200)) v_object 0xd30a84a0 ref 0 pages 1070 lock type devfs: EXCL by thread 0xcb2e58d0 (pid 64002, umount, tid 100175) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0cdedb7 at ffs_sync+0x467 #5 0xc0b20dd8 at dounmount+0x3f8 #6 0xc0b214c1 at sys_unmount+0x421 #7 0xc0e01a2e at syscall+0x2fe #8 0xc0debe91 at Xint0x80_syscall+0x21 dev md5a 20120518 07:50:44 all: memguard.sh 20120518 07:50:52 all: inversion.sh 20120518 08:20:56 all: jail.sh 20120518 08:20:57 all: zfs2.sh ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 5 ZFS storage pool version 28 20120518 08:32:06 all: suj7.sh Failed to find journal. Use tunefs to create one Failed to start journal: 2 20120518 08:32:07 all: posix_fadvise.sh 20120518 08:32:09 all: malloc3.sh 20120518 09:03:13 all: suj22.sh 20120518 09:04:14 all: nfs4.sh May 18 09:04:15 x4 mountd[1009]: bad exports list line /mnt 20120518 09:05:16 all: suj20.sh 20120518 09:06:40 all: lockf3.sh May 18 09:11:33 x4 mountd[1009]: bad exports list line /mnt May 18 09:13:58 x4 mountd[1009]: bad exports list line /mnt May 18 09:19:29 x4 mountd[1009]: bad exports list line /mnt May 18 11:04:38 x4 mountd[1009]: bad exports list line /mnt May 18 11:08:46 x4 last message repeated 6 times May 18 11:09:07 x4 su: BAD SU pho to root on /dev/pts/2 May 18 11:09:33 x4 mountd[1009]: bad exports list line /mnt May 18 11:10:34 x4 mountd[1009]: bad exports list line /mnt May 18 11:15:39 x4 su: pho to root on /dev/pts/2 May 18 11:16:26 x4 mountd[1009]: bad exports list line /mnt 20120518 11:24:41 all: mac_chkexec.sh 20120518 11:24:44 all: devfs2.sh 20120518 11:24:45 all: zfs4.sh 20120518 11:35:28 all: statfs.sh 20120518 11:35:46 all: context.sh 20120518 11:37:02 all: suj21.sh 20120518 11:38:27 all: ffs_blkfree.sh 20120518 11:38:48 all: revoke.sh 20120518 11:38:49 all: rename4.sh 20120518 11:38:50 all: jail3.sh 20120518 11:38:51 all: msdos4.sh 20120518 11:59:03 all: suj2.sh 20120518 11:59:06 all: uio.sh 20120518 11:59:08 all: nfs4.sh May 18 11:59:09 x4 mountd[1009]: bad exports list line /mnt 20120518 12:01:52 all: nullfs5.sh 20120518 12:14:12 all: suj4.sh 20120518 13:14:58 all: zfs2.sh Stopping inetd. Waiting for PIDS: 1228. Shutting down local packages:. Stopping cron. Waiting for PIDS: 1158. Stopping sshd. Waiting for PIDS: 1147. Stopping moused. Waiting for PIDS: 1115. Stopping ntpd. Waiting for PIDS: 1091. Stopping nfsd. Waiting for PIDS: 1031 1035. Stopping mountd. Waiting for PIDS: 1009. Stopping nfsuserd. Waiting for PIDS: 1025 1026 1027 1028 1029. rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered Stopping rpcbind. Waiting for PIDS: 976. Stopping watchdogd. Waiting for PIDS: 120390 second watchdog timeout expired. Shutdown terminated. Fri May 18 13:35:54 CEST 2012 May 18 13:35:54 x4 init: /bin/shMay 18 13:35:55 x4 syslogd: exiting on signal 15 May 18 13:36:14 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 63116 1 0 20 0 13972 2104 wait Ss u0 0:00.01 -sh (sh) 0 63117 63116 0 20 0 13736 1176 - R+ u0 0:00.00 ps -l 0 62311 1 0 20 0 1636 624 tmpfs DL 0- 0:00.00 /tmp/dl 0 62312 62311 0 20 0 1636 624 tmpfs DL 0- 0:00.00 /tmp/dl 0 62348 1 0 20 0 13720 960 tmpfs D 2- 0:00.00 procstat -f 62312 # ~KDB: enter: Break to debugger [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> show mount 0xca77fb70 /dev/ad4s1a on / (ufs) 0xca780000 devfs on /dev (devfs) 0xcab205b8 /dev/ad4s1f on /home (ufs) 0xcab20000 /dev/ad4s1d on /usr (ufs) 0xca780b70 /dev/ad4s1e on /var (ufs) 0xca780894 procfs on /proc (procfs) 0xd00e2000 /dev/label/tmp on /tmp (ufs) 0xcaa842dc tmpfs on /mnt (tmpfs) More info: show mount db> show mount 0xcaa842dc 0xcaa842dc tmpfs on /mnt (tmpfs) mnt_flag = EXPORTED, LOCAL mnt_kern_flag = MPSAFE mnt_opt = fstype, fspath, from, errmsg mnt_stat = { version=537068824 type=135 flags=0x0000000000001100 bsize=4096 iosize=4096 blocks=4760640 bfree=4760603 bavail=4760603 files=2147483647 ffree=2147483642 syncwrites=0 asyncwrites=0 syncreads=0 asyncreads=0 namemax=255 owner=0 fsid=[-2025717884, 135] } mnt_cred = { uid=0 ruid=0 } mnt_ref = 8 mnt_gen = 361 mnt_nvnodelistsize = 6 mnt_activevnodelistsize = 6 mnt_writeopcount = 2 mnt_maxsymlinklen = 0 mnt_iosize_max = 65536 mnt_hashseed = 261131325 mnt_secondary_writes = 0 mnt_secondary_accwrites = 0 mnt_gjprovider = NULL List of active vnodes vnode 0xce4999d8: tag tmpfs, type VREG usecount 3, writecount 2, refcount 4 mountedhere 0 flags (VI(0x200)) v_object 0xcac722e4 ref 2 pages 1 lock type tmpfs: UNLOCKED #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b2bcbb at vget+0xbb #5 0xc0d04f7b at vm_fault_hold+0x92b #6 0xc0d066ab at vm_fault+0x8b #7 0xc0e01519 at trap_pfault+0x1b9 #8 0xc0e02531 at trap+0x5f1 #9 0xc0debdfc at calltrap+0x6 #10 0xd48cefa9 at tmpfs_write+0x6a9 #11 0xc0e29694 at VOP_WRITE_APV+0x174 #12 0xc0b3a57c at vn_write+0x2bc #13 0xc0b3809a at vn_io_fault+0x7a #14 0xc0ad7de5 at dofilewrite+0x95 #15 0xc0ad8458 at kern_writev+0x68 #16 0xc0ad856f at sys_write+0x4f #17 0xc0e01a2e at syscall+0x2fe tag VT_TMPFS, tmpfs_node 0xccc57ee0, flags 0x0, links 1 mode 0700, owner 0, group 0, size 32768, status 0x2 vnode 0xd1cb9438: tag tmpfs, type VREG usecount 3, writecount 2, refcount 4 mountedhere 0 flags (VI(0x200)) v_object 0xce854de0 ref 2 pages 1 lock type tmpfs: UNLOCKED #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b2bcbb at vget+0xbb #5 0xc0d04f7b at vm_fault_hold+0x92b #6 0xc0d066ab at vm_fault+0x8b #7 0xc0e01519 at trap_pfault+0x1b9 #8 0xc0e02531 at trap+0x5f1 #9 0xc0debdfc at calltrap+0x6 #10 0xd48cefa9 at tmpfs_write+0x6a9 #11 0xc0e29694 at VOP_WRITE_APV+0x174 #12 0xc0b3a57c at vn_write+0x2bc #13 0xc0b3809a at vn_io_fault+0x7a #14 0xc0ad7de5 at dofilewrite+0x95 #15 0xc0ad8458 at kern_writev+0x68 #16 0xc0ad856f at sys_write+0x4f #17 0xc0e01a2e at syscall+0x2fe tag VT_TMPFS, tmpfs_node 0xcccd7000, flags 0x0, links 1 mode 0700, owner 0, group 0, size 32768, status 0x2 vnode 0xd2b089d8: tag tmpfs, type VREG usecount 3, writecount 2, refcount 6 mountedhere 0 flags (VI(0x200)) v_object 0xd1f1c1bc ref 2 pages 1 lock type tmpfs: EXCL by thread 0xcc1a62f0 (pid 62311, dl, tid 5171748) with exclusive waiters pending #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b3a436 at vn_write+0x176 #5 0xc0b3809a at vn_io_fault+0x7a #6 0xc0ad7de5 at dofilewrite+0x95 #7 0xc0ad8458 at kern_writev+0x68 #8 0xc0ad856f at sys_write+0x4f #9 0xc0e01a2e at syscall+0x2fe #10 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xcccd7088, flags 0x0, links 1 mode 0700, owner 0, group 0, size 40960, status 0x0 vnode 0xd203f5a0: tag tmpfs, type VREG usecount 5, writecount 2, refcount 9 mountedhere 0 flags (VI(0x200)) v_object 0xce3c58ac ref 2 pages 1 lock type tmpfs: EXCL by thread 0xcb7db5e0 (pid 62312, dl, tid 100610) with exclusive waiters pending #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b3a436 at vn_write+0x176 #5 0xc0b3809a at vn_io_fault+0x7a #6 0xc0ad7de5 at dofilewrite+0x95 #7 0xc0ad8458 at kern_writev+0x68 #8 0xc0ad856f at sys_write+0x4f #9 0xc0e01a2e at syscall+0x2fe #10 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xcccd7110, flags 0x0, links 1 mode 0700, owner 0, group 0, size 40960, status 0x0 vnode 0xce30e870: tag tmpfs, type VDIR usecount 5, writecount 0, refcount 5 mountedhere 0 flags (VV_ROOT|VI(0x200)) v_object 0xcc200b90 ref 0 pages 0 lock type tmpfs: EXCL by thread 0xcab182f0 (pid 62380, ls, tid 100156) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b1b1b6 at lookup+0x106 #5 0xc0b1c7ff at namei+0x6ff #6 0xc0b2f62f at kern_statat_vnhook+0xbf #7 0xc0b2f82c at kern_statat+0x3c #8 0xc0b2f866 at kern_lstat+0x36 #9 0xc0b2f90f at sys_lstat+0x2f #10 0xc0e01a2e at syscall+0x2fe #11 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xccc57000, flags 0x0, links 2 mode 0777, owner 0, group 0, size 80, status 0x2 vnode 0xd0708438: tag syncer, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI(0x200)) lock type syncer: UNLOCKED #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b2bf02 at sync_vnode+0x142 #5 0xc0b2c282 at sched_sync+0x282 #6 0xc0a575d8 at fork_exit+0xb8 #7 0xc0debea4 at fork_trampoline+0x8 List of inactive vnodes db> run pho db:0:pho> bt Tracing pid 11 tid 100005 td 0xc7ddf2f0 kdb_enter(c0e47e69,c0fd7bbe,c7b2cb9c,c08fb9bd,c7da7200,...) at kdb_enter+0x3a kdb_break(c7da7200,c0e1cb9e,1,c119e140,c7b2cbd0,...) at kdb_break+0x28 uart_intr(c7da7200,c133d2f8,c7b2cbdc,c0e1cb9e,0,...) at uart_intr+0x12d intr_event_handle(c7daab80,c7b2cc08,c136f580,0,c9470800,...) at intr_event_handle+0x76 intr_execute_handlers(c7e210d0,c7b2cc08,2,c7b2cc48,c0dec1f5,...) at intr_execute_handlers+0x49 lapic_handle_intr(30,c7b2cc08) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0de7765, esp = 0xc7b2cc48, ebp = 0xc7b2cc48 --- acpi_cpu_c1(c7b2cc64,c1153730,2,317,11f44,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b2cc98,c0df49c9,0,2,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,2,0,2,c7b2cce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b2ccc0,c0fd5c2c,a41,c7ddf2f0,...) at cpu_idle+0x89 sched_idletd(0,c7b2cd28,c0fcda70,3d8,c7ddd864,...) at sched_idletd+0x235 fork_exit(c0ab0170,0,c7b2cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b2cd60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x6d2c00 curthread = 0xc7ddf8d0: pid 11 "idle: cpu0" curpcb = 0xc7b26d80 fpcurthread = none idlethread = 0xc7ddf8d0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x69c3c00 curthread = 0xc7ddf5e0: pid 11 "idle: cpu1" curpcb = 0xc7b29d80 fpcurthread = none idlethread = 0xc7ddf5e0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x69c6c00 curthread = 0xc7ddf2f0: pid 11 "idle: cpu2" curpcb = 0xc7b2cd80 fpcurthread = none idlethread = 0xc7ddf2f0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x69c9c00 curthread = 0xc7ddf000: pid 11 "idle: cpu3" curpcb = 0xc7b2fd80 fpcurthread = none idlethread = 0xc7ddf000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 62380 (ls) thread 0xcab182f0 (100156) exclusive lockmgr tmpfs (tmpfs) r = 0 (0xce30e8c8) locked @ kern/vfs_lookup.c:542 Process 62312 (dl) thread 0xcb7db5e0 (100610) exclusive lockmgr tmpfs (tmpfs) r = 0 (0xd203f5f8) locked @ kern/vfs_vnops.c:652 Process 62311 (dl) thread 0xcc1a62f0 (5171748) exclusive lockmgr tmpfs (tmpfs) r = 0 (0xd2b08a30) locked @ kern/vfs_vnops.c:652 db:0:alllocks> show lockedvnods Locked vnodes 0xce30e870: tag tmpfs, type VDIR usecount 5, writecount 0, refcount 5 mountedhere 0 flags (VV_ROOT|VI(0x200)) v_object 0xcc200b90 ref 0 pages 0 lock type tmpfs: EXCL by thread 0xcab182f0 (pid 62380, ls, tid 100156) #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b1b1b6 at lookup+0x106 #5 0xc0b1c7ff at namei+0x6ff #6 0xc0b2f62f at kern_statat_vnhook+0xbf #7 0xc0b2f82c at kern_statat+0x3c #8 0xc0b2f866 at kern_lstat+0x36 #9 0xc0b2f90f at sys_lstat+0x2f #10 0xc0e01a2e at syscall+0x2fe #11 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xccc57000, flags 0x0, links 2 mode 0777, owner 0, group 0, size 80, status 0x2 0xd203f5a0: tag tmpfs, type VREG usecount 5, writecount 2, refcount 9 mountedhere 0 flags (VI(0x200)) v_object 0xce3c58ac ref 2 pages 1 lock type tmpfs: EXCL by thread 0xcb7db5e0 (pid 62312, dl, tid 100610) with exclusive waiters pending #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b3a436 at vn_write+0x176 #5 0xc0b3809a at vn_io_fault+0x7a #6 0xc0ad7de5 at dofilewrite+0x95 #7 0xc0ad8458 at kern_writev+0x68 #8 0xc0ad856f at sys_write+0x4f #9 0xc0e01a2e at syscall+0x2fe #10 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xcccd7110, flags 0x0, links 1 mode 0700, owner 0, group 0, size 40960, status 0x0 0xd2b089d8: tag tmpfs, type VREG usecount 3, writecount 2, refcount 6 mountedhere 0 flags (VI(0x200)) v_object 0xd1f1c1bc ref 2 pages 1 lock type tmpfs: EXCL by thread 0xcc1a62f0 (pid 62311, dl, tid 5171748) with exclusive waiters pending #0 0xc0a6cdae at __lockmgr_args+0xd6e #1 0xc0b15e25 at vop_stdlock+0x65 #2 0xc0e28853 at VOP_LOCK1_APV+0xf3 #3 0xc0b39748 at _vn_lock+0x78 #4 0xc0b3a436 at vn_write+0x176 #5 0xc0b3809a at vn_io_fault+0x7a #6 0xc0ad7de5 at dofilewrite+0x95 #7 0xc0ad8458 at kern_writev+0x68 #8 0xc0ad856f at sys_write+0x4f #9 0xc0e01a2e at syscall+0x2fe #10 0xc0debe91 at Xint0x80_syscall+0x21 tag VT_TMPFS, tmpfs_node 0xcccd7088, flags 0x0, links 1 mode 0700, owner 0, group 0, size 40960, status 0x0 db:0:lockedvnods> show mount 0xca77fb70 /dev/ad4s1a on / (ufs) 0xca780000 devfs on /dev (devfs) 0xcab205b8 /dev/ad4s1f on /home (ufs) 0xcab20000 /dev/ad4s1d on /usr (ufs) 0xca780b70 /dev/ad4s1e on /var (ufs) 0xca780894 procfs on /proc (procfs) 0xd00e2000 /dev/label/tmp on /tmp (ufs) 0xcaa842dc tmpfs on /mnt (tmpfs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 63116 1 63116 0 Ss+ ttyin 0xc7ef8e70 sh 62380 1 1203 0 D tmpfs 0xd203f5f8 ls 62348 1 62348 0 D+ tmpfs 0xd203f5f8 procstat 62312 62311 62301 0 DL+ tmpfs 0xd2b08a30 dl 62311 1 62301 0 DL+ tmpfs 0xd203f5f8 dl 45311 0 0 0 DL (threaded) [zfskern] 5171774 D l2arc_fe 0xd75ff284 [l2arc_feed_thread] 5171251 D arc_recl 0xd75f00fc [arc_reclaim_thread] 76075 0 0 0 DL crypto_r 0xd6c21ccc [crypto returns] 76074 0 0 0 DL crypto_w 0xd6c21ca8 [crypto] 19 0 0 0 DL sdflush 0xc130e6c0 [softdepflush] 18 0 0 0 DL vlruwt 0xca6da2cc [vnlru] 17 0 0 0 DL syncer 0xc13069b4 [syncer] 16 0 0 0 DL psleep 0xc1306648 [bufdaemon] 9 0 0 0 DL pgzero 0xc131065c [pagezero] 8 0 0 0 DL psleep 0xc13101c8 [vmdaemon] 7 0 0 0 DL psleep 0xc1310190 [pagedaemon] 6 0 0 0 DL ccb_scan 0xc1153154 [xpt_thrd] 5 0 0 0 DL waiting_ 0xc1309ed8 [sctp_iterator] 4 0 0 0 DL - 0xc947163c [fdc0] 3 0 0 0 SL - 0xc952b000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100059 D - 0xc94fdd34 [usbus5] 100058 D - 0xc94fdd04 [usbus5] 100057 D - 0xc94fdcd4 [usbus5] 100056 D - 0xc94fdca4 [usbus5] 100054 D - 0xc94eeb5c [usbus4] 100053 D - 0xc94eeb2c [usbus4] 100052 D - 0xc94eeafc [usbus4] 100051 D - 0xc94eeacc [usbus4] 100050 D - 0xc94e9b5c [usbus3] 100049 D - 0xc94e9b2c [usbus3] 100048 D - 0xc94e9afc [usbus3] 100047 D - 0xc94e9acc [usbus3] 100046 D - 0xc94e4b5c [usbus2] 100045 D - 0xc94e4b2c [usbus2] 100044 D - 0xc94e4afc [usbus2] 100043 D - 0xc94e4acc [usbus2] 100041 D - 0xc94dcb5c [usbus1] 100040 D - 0xc94dcb2c [usbus1] 100039 D - 0xc94dcafc [usbus1] 100038 D - 0xc94dcacc [usbus1] 100036 D - 0xc94d4b5c [usbus0] 100035 D - 0xc94d4b2c [usbus0] 100034 D - 0xc94d4afc [usbus0] 100033 D - 0xc94d4acc [usbus0] 2 0 0 0 DL ctl_work 0xc7fc4000 [ctl_thrd] 14 0 0 0 DL - 0xc118f5e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc118c6a8 [g_down] 100014 D - 0xc118c6a4 [g_up] 100013 D - 0xc118c69c [g_event] 12 0 0 0 WL (threaded) [intr] 100067 I [irq12: psm0] 100066 I [irq1: atkbd0] 100064 I [swi0: uart] 100061 I [irq20: fwohci0] 100060 I [irq14: ata0] 100055 I [irq19: ehci0] 100042 I [irq18: ohci2 ohci4] 100037 I [irq17: ohci1 ohci3] 100032 I [irq16: hdac1 ohci0] 100031 I [irq22: ahci0] 100030 I [irq257: re0] 100029 I [irq256: hdac0] 100025 I [swi5: +] 100020 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc7dddb30 [init] 10 0 0 0 DL audit_wo 0xc130de00 [audit] 0 0 0 0 DLs (threaded) [kernel] 5171773 D - 0xcceac840 [system_taskq_3] 5171772 D - 0xcceac840 [system_taskq_2] 5171771 D - 0xcceac840 [system_taskq_1] 5171770 D - 0xcceac840 [system_taskq_0] 100069 D - 0xc118f5e4 [deadlkres] 100068 D - 0xc7f0bdc0 [mca taskq] 100062 D - 0xc7f0be00 [fw0_taskq] 100027 D - 0xc7f0be40 [thread taskq] 100026 D - 0xc7f0be80 [ffs_trim taskq] 100024 D - 0xc7f0c040 [kqueue taskq] 100023 D - 0xc7f0c080 [acpi_task_2] 100022 D - 0xc7f0c080 [acpi_task_1] 100021 D - 0xc7f0c080 [acpi_task_0] 100016 D - 0xc7ef0000 [firmware taskq] 100000 D sched 0xc118c860 [swapper] db:0:ps> allt Tracing command sh pid 63116 tid 100134 td 0xcb1a35e0 sched_switch(cb1a35e0,0,104,1b3,f284aa1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0a75fdb,...) at mi_switch+0x243 sleepq_switch(cb1a35e0,0,c0fd9600,1ac,0,...) at sleepq_switch+0x1a7 sleepq_catch_signals(c0a74eea,c7ef8e04,0,c0fd1a67,cb1a35e0,...) at sleepq_catch_signals+0xfc sleepq_wait_sig(c7ef8e70,0,f284aae8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7ef8e70,c7ef8e04,c0fddd98,527,0,...) at _cv_wait_sig+0x260 tty_wait(c7ef8e00,c7ef8e70,f284ac3c,1,0,...) at tty_wait+0x71 ttydisc_read(c7ef8e00,f284ac3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c7dc6000,f284ac3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cba81bd0,f284ac3c,c7dd9b80,0,cb1a35e0,...) at devfs_read_f+0x7e dofileread(f284ac3c,ffffffff,ffffffff,0,f284ac18,...) at dofileread+0x9e kern_readv(cb1a35e0,0,f284ac3c,f284ac5c,1,...) at kern_readv+0x68 sys_read(cb1a35e0,f284acec,c102715c,c0e38372,c1028770,...) at sys_read+0x4f syscall(f284ad28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281ed7c3, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 --- Tracing command ls pid 62380 tid 100156 td 0xcab182f0 sched_switch(cab182f0,0,104,1b3,f28916f0,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cac037c0,...) at mi_switch+0x243 sleepq_switch(cab182f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d203f5f8,60,d48d7014,4,0,...) at sleepq_wait+0x6b __lockmgr_args(d203f5f8,80100,d203f664,0,0,...) at __lockmgr_args+0xc86 vop_stdlock(f2891840,c0ad5b8c,c7d8a2a8,c,c12c8630,...) at vop_stdlock+0x65 VOP_LOCK1_APV(d48d80a0,f2891840,cab183a4,c114b840,d203f5a0,...) at VOP_LOCK1_APV+0xf3 _vn_lock(d203f5a0,80100,c0fe3751,870,eb,...) at _vn_lock+0x78 vget(d203f5a0,80100,cab182f0,187,ccc57000,...) at vget+0xbb tmpfs_alloc_vp(caa842dc,cccd7110,80000,f2891af8,f2891900,...) at tmpfs_alloc_vp+0xb7 tmpfs_lookup(f2891964,c102e093,c0b15f0a,ce30e8c8,f2891954,...) at tmpfs_lookup+0x1e4 VOP_CACHEDLOOKUP_APV(d48d80a0,f2891964,f2891b0c,0,0,...) at VOP_CACHEDLOOKUP_APV+0x100 vfs_cache_lookup(f2891a0c,c0fe44a8,c0b2accf,d48d80a0,f28919a0,...) at vfs_cache_lookup+0xe6 VOP_LOOKUP_APV(d48d80a0,f2891a0c,f2891b0c,21e,c0ad4d0f,...) at VOP_LOOKUP_APV+0x108 lookup(f2891acc,c0fe2b2c,10e,dc,4c,...) at lookup+0x6eb namei(f2891acc,ce30e8c8,c12c8630,cab183a4,c12c8630,...) at namei+0x6ff kern_statat_vnhook(cab182f0,200,ffffff9c,881f2f8,0,...) at kern_statat_vnhook+0xbf kern_statat(cab182f0,200,ffffff9c,881f2f8,0,...) at kern_statat+0x3c kern_lstat(cab182f0,881f2f8,0,f2891bfc,0,...) at kern_lstat+0x36 sys_lstat(cab182f0,f2891cec,c102715c,c0fdc7b9,c0ab1df7,...) at sys_lstat+0x2f syscall(f2891d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (190, FreeBSD ELF32, sys_lstat), eip = 0x281b89f3, esp = 0xbfbfe55c, ebp = 0xbfbfe5e8 --- Tracing command procstat pid 62348 tid 100636 td 0xcb7d72f0 sched_switch(cb7d72f0,0,104,1b3,f2fae71c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cac037c0,...) at mi_switch+0x243 sleepq_switch(cb7d72f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d203f5f8,60,d48d7014,4,0,...) at sleepq_wait+0x6b __lockmgr_args(d203f5f8,200400,d203f664,0,0,...) at __lockmgr_args+0xc86 vop_stdlock(f2fae86c,c13066fc,c12cedb8,cb7d73a4,c12ceda8,...) at vop_stdlock+0x65 VOP_LOCK1_APV(d48d80a0,f2fae86c,0,c114b840,d203f5a0,...) at VOP_LOCK1_APV+0xf3 _vn_lock(d203f5a0,200400,c0fe1e5b,4df,c0fd1a67,...) at _vn_lock+0x78 vn_vptocnp_locked(f2fae930,c0fe1e5b,513,f2fae920,c0a9085e,...) at vn_vptocnp_locked+0x31b vn_fullpath1(cd154c00,f2faeab0,3ff,47f,d203f5a0,...) at vn_fullpath1+0xc0 vn_fullpath(cb7d72f0,d203f5a0,f2faeab0,f2faeaac,3,...) at vn_fullpath+0xb6 export_fd_for_sysctl(3,3,8000,0,0,...) at export_fd_for_sysctl+0xc6 sysctl_kern_proc_filedesc(c11052a0,f2faec00,1,f2faeb88,f2faeb88,...) at sysctl_kern_proc_filedesc+0x720 sysctl_root(f2faeb88,0,c0fd4878,656,cb7d72f0,...) at sysctl_root+0x20e userland_sysctl(cb7d72f0,f2faebf4,4,0,bfbfe158,...) at userland_sysctl+0x17a sys___sysctl(cb7d72f0,f2faecec,c102715c,c0fdc914,c1028770,...) at sys___sysctl+0x94 syscall(f2faed28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (202, FreeBSD ELF32, sys___sysctl), eip = 0x28196e87, esp = 0xbfbfe0dc, ebp = 0xbfbfe108 --- Tracing command dl pid 62312 tid 100610 td 0xcb7db5e0 sched_switch(cb7db5e0,0,104,1b3,f2f60440,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cb4d98c0,...) at mi_switch+0x243 sleepq_switch(cb7db5e0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d2b08a30,60,d48d7014,4,0,...) at sleepq_wait+0x6b __lockmgr_args(d2b08a30,200501,d2b08a9c,0,0,...) at __lockmgr_args+0xc86 vop_stdlock(f2f60590,0,f2f60584,c0ad375a,eb,...) at vop_stdlock+0x65 VOP_LOCK1_APV(d48d80a0,f2f60590,d2b08a9c,c114b840,d2b089d8,...) at VOP_LOCK1_APV+0xf3 _vn_lock(d2b089d8,200501,c0fe3751,870,c0d04094,...) at _vn_lock+0x78 vget(d2b089d8,200401,cb7db5e0,96,f2f6075c,...) at vget+0xbb vm_fault_hold(caeda9f8,281b5000,1,0,0,...) at vm_fault_hold+0x9ed vm_fault(caeda9f8,281b5000,1,0,281b5000,...) at vm_fault+0x8b trap_pfault(c7d87ba8,c7d85370,4,c12d74c0,cb7be2cc,...) at trap_pfault+0x1b9 trap(f2f60920) at trap+0x5f1 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0dff312, esp = 0xf2f60960, ebp = 0xf2f609a8 --- copyin(f2f609fc,0,1000,f2f60c3c,0,...) at copyin+0x32 tmpfs_write(f2f60ae0,c102e8a3,80400,d203f5a0,f2f60a98,...) at tmpfs_write+0x6a9 VOP_WRITE_APV(d48d80a0,f2f60ae0,d203f5a0,28c,0,...) at VOP_WRITE_APV+0x174 vn_write(d4912428,f2f60c3c,ca7cc900,0,cb7db5e0,...) at vn_write+0x2bc vn_io_fault(d4912428,f2f60c3c,ca7cc900,0,cb7db5e0,...) at vn_io_fault+0x7a dofilewrite(f2f60c3c,ffffffff,ffffffff,0,f2f60c18,...) at dofilewrite+0x95 kern_writev(cb7db5e0,3,f2f60c3c,f2f60c5c,1,...) at kern_writev+0x68 sys_write(cb7db5e0,f2f60cec,c102715c,c0fdc437,c1028770,...) at sys_write+0x4f syscall(f2f60d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281717a3, esp = 0xbfbfe7fc, ebp = 0xbfbfe818 --- Tracing command dl pid 62311 tid 5171748 td 0xcc1a62f0 sched_switch(cc1a62f0,0,104,1b3,f3399440,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cac037c0,...) at mi_switch+0x243 sleepq_switch(cc1a62f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d203f5f8,60,d48d7014,4,0,...) at sleepq_wait+0x6b __lockmgr_args(d203f5f8,200501,d203f664,0,0,...) at __lockmgr_args+0xc86 vop_stdlock(f3399590,0,f3399584,c0ad375a,eb,...) at vop_stdlock+0x65 VOP_LOCK1_APV(d48d80a0,f3399590,d203f664,c114b840,d203f5a0,...) at VOP_LOCK1_APV+0xf3 _vn_lock(d203f5a0,200501,c0fe3751,870,c0d04094,...) at _vn_lock+0x78 vget(d203f5a0,200401,cc1a62f0,96,f339975c,...) at vget+0xbb vm_fault_hold(cad371d0,281b7000,1,0,0,...) at vm_fault_hold+0x9ed vm_fault(cad371d0,281b7000,1,0,281b7000,...) at vm_fault+0x8b trap_pfault(f3399858,c0dff0c9,1,c7d82930,ca9e6b30,...) at trap_pfault+0x1b9 trap(f3399920) at trap+0x5f1 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0dff312, esp = 0xf3399960, ebp = 0xf33999a8 --- copyin(f33999fc,0,1000,f3399c3c,0,...) at copyin+0x32 tmpfs_write(f3399ae0,c102e8a3,80400,d2b089d8,f3399a98,...) at tmpfs_write+0x6a9 VOP_WRITE_APV(d48d80a0,f3399ae0,d2b089d8,28c,0,...) at VOP_WRITE_APV+0x174 vn_write(cb8040e0,f3399c3c,ca7cc900,0,cc1a62f0,...) at vn_write+0x2bc vn_io_fault(cb8040e0,f3399c3c,ca7cc900,0,cc1a62f0,...) at vn_io_fault+0x7a dofilewrite(f3399c3c,ffffffff,ffffffff,0,f3399c18,...) at dofilewrite+0x95 kern_writev(cc1a62f0,4,f3399c3c,f3399c5c,1,...) at kern_writev+0x68 sys_write(cc1a62f0,f3399cec,c102715c,c0fcdafc,c0ab1df7,...) at sys_write+0x4f syscall(f3399d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281717a3, esp = 0xbfbfe7fc, ebp = 0xbfbfe818 --- Tracing command zfskern pid 45311 tid 5171774 td 0xcc4048d0 sched_switch(cc4048d0,0,104,1b3,f33e7b28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,8,...) at mi_switch+0x243 sleepq_switch(cc4048d0,0,c0fd9600,291,cc4048d0,...) at sleepq_switch+0x1a7 sleepq_timedwait(d75ff284,0,f33e7bb4,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(d75ff284,d75ff270,3e8,0,0,...) at _cv_timedwait+0x270 l2arc_feed_thread(0,f33e7d28,c0fcda70,3d8,d08b52cc,...) at l2arc_feed_thread+0x204 fork_exit(d74d08c0,0,f33e7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf33e7d60, ebp = 0 --- Tracing command zfskern pid 45311 tid 5171251 td 0xcabf9000 sched_switch(cabf9000,0,104,1b3,f2c3bbdc,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,8,...) at mi_switch+0x243 sleepq_switch(cabf9000,0,c0fd9600,291,cabf9000,...) at sleepq_switch+0x1a7 sleepq_timedwait(d75f00fc,0,f2c3bc68,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(d75f00fc,d75f00e8,3e8,0,24f8ef40,...) at _cv_timedwait+0x270 arc_reclaim_thread(0,f2c3bd28,c0fcda70,3d8,d08b52cc,...) at arc_reclaim_thread+0x28e fork_exit(d74d0460,0,f2c3bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf2c3bd60, ebp = 0 --- Tracing command crypto returns pid 76075 tid 3114027 td 0xcbae25e0 sched_switch(cbae25e0,0,104,1b3,f2ef4c0c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,ce83b900,...) at mi_switch+0x243 sleepq_switch(cbae25e0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d6c21ccc,6c,d6c20714,0,0,...) at sleepq_wait+0x6b _sleep(d6c21ccc,d6c21cdc,6c,d6c20714,0,...) at _sleep+0x39c crypto_ret_proc(0,f2ef4d28,c0fcda70,3d8,cb3fc598,...) at crypto_ret_proc+0x18e fork_exit(d6c03a00,0,f2ef4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf2ef4d60, ebp = 0 --- Tracing command crypto pid 76074 tid 5171482 td 0xcc1408d0 sched_switch(cc1408d0,0,104,1b3,f32d6c18,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cac00a80,...) at mi_switch+0x243 sleepq_switch(cc1408d0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(d6c21ca8,6c,d6c20963,0,0,...) at sleepq_wait+0x6b _sleep(d6c21ca8,d6c21cb8,6c,d6c20963,0,...) at _sleep+0x39c crypto_proc(0,f32d6d28,c0fcda70,3d8,cc0aa864,...) at crypto_proc+0x1b0 fork_exit(d6c04dc0,0,f32d6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf32d6d60, ebp = 0 --- Tracing command softdepflush pid 19 tid 100078 td 0xca6e78d0 sched_switch(ca6e78d0,0,104,1b3,ee324c10,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(ca6e78d0,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c130e6c0,54,c100b791,0,0,...) at sleepq_timedwait+0x6b _sleep(c130e6c0,c130e680,54,c100b791,3e8,...) at _sleep+0x36a softdep_flush(0,ee324d28,c0fcda70,3d8,ca6da000,...) at softdep_flush+0x259 fork_exit(c0cd9ed0,0,ee324d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee324d60, ebp = 0 --- Tracing command vnlru pid 18 tid 100077 td 0xca6e7bc0 sched_switch(ca6e7bc0,0,104,1b3,ee321be4,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(ca6e7bc0,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(ca6da2cc,60,c0fe4775,0,0,...) at sleepq_timedwait+0x6b _sleep(ca6da2cc,c1306974,260,c0fe4775,3e8,...) at _sleep+0x36a vnlru_proc(0,ee321d28,c0fcda70,3d8,ca6da2cc,...) at vnlru_proc+0xe7 fork_exit(c0b2cce0,0,ee321d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee321d60, ebp = 0 --- Tracing command syncer pid 17 tid 100076 td 0xc94f88d0 sched_switch(c94f88d0,0,104,1b3,ee31ebec,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd1a67,...) at mi_switch+0x243 sleepq_switch(c94f88d0,0,c0fd9600,291,c94f88d0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c13069b4,0,ee31ec78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c13069b4,c13069a0,3e8,784,4e20,...) at _cv_timedwait+0x270 sched_sync(0,ee31ed28,c0fcda70,3d8,ca6da598,...) at sched_sync+0x595 fork_exit(c0b2c000,0,ee31ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee31ed60, ebp = 0 --- Tracing command bufdaemon pid 16 tid 100075 td 0xc94f8bc0 sched_switch(c94f8bc0,0,104,1b3,ee31bc1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c94f8bc0,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c1306648,54,c0fe14d2,0,0,...) at sleepq_timedwait+0x6b _sleep(c1306648,c1306650,54,c0fe14d2,3e8,...) at _sleep+0x36a buf_daemon(0,ee31bd28,c0fcda70,3d8,ca6da864,...) at buf_daemon+0x175 fork_exit(c0b0ece0,0,ee31bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee31bd60, ebp = 0 --- Tracing command pagezero pid 9 tid 100074 td 0xc9623000 sched_switch(c9623000,0,104,1b3,ee318c20,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c9623000,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c131065c,0,c1012981,0,0,...) at sleepq_timedwait+0x6b _sleep(c131065c,c130f000,0,c1012981,493e0,...) at _sleep+0x36a vm_pagezero(0,ee318d28,c0fcda70,3d8,ca6dab30,...) at vm_pagezero+0xdc fork_exit(c0d21070,0,ee318d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee318d60, ebp = 0 --- Tracing command vmdaemon pid 8 tid 100073 td 0xc96232f0 sched_switch(c96232f0,0,104,1b3,ee315be4,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbfd80,...) at mi_switch+0x243 sleepq_switch(c96232f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c13101c8,74,c0fe14d2,0,0,...) at sleepq_wait+0x6b _sleep(c13101c8,c13101cc,74,c0fe14d2,0,...) at _sleep+0x39c vm_daemon(0,ee315d28,c0fcda70,3d8,ca6db000,...) at vm_daemon+0x59 fork_exit(c0d1a630,0,ee315d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee315d60, ebp = 0 --- Tracing command pagedaemon pid 7 tid 100072 td 0xc96235e0 sched_switch(c96235e0,0,104,1b3,ee312b6c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c96235e0,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c1310190,54,c0fe14d2,0,0,...) at sleepq_timedwait+0x6b _sleep(c1310190,c130f000,54,c0fe14d2,1388,...) at _sleep+0x36a vm_pageout(0,ee312d28,c0fcda70,3d8,c7dde2cc,...) at vm_pageout+0x2c7 fork_exit(c0d1b6b0,0,ee312d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee312d60, ebp = 0 --- Tracing command xpt_thrd pid 6 tid 100071 td 0xc96238d0 sched_switch(c96238d0,0,104,1b3,ee18fc1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbfd00,...) at mi_switch+0x243 sleepq_switch(c96238d0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c1153154,5c,c0e2dd9e,0,0,...) at sleepq_wait+0x6b _sleep(c1153154,c1153198,5c,c0e2dd9e,0,...) at _sleep+0x39c xpt_scanner_thread(0,ee18fd28,c0fcda70,3d8,c7dde598,...) at xpt_scanner_thread+0x47 fork_exit(c04a7790,0,ee18fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee18fd60, ebp = 0 --- Tracing command sctp_iterator pid 5 tid 100070 td 0xc9623bc0 sched_switch(c9623bc0,0,104,1b3,ee18cc24,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbfcc0,...) at mi_switch+0x243 sleepq_switch(c9623bc0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c1309ed8,0,c0ff8c7f,0,0,...) at sleepq_wait+0x6b _sleep(c1309ed8,c1309ea0,0,c0ff8c7f,0,...) at _sleep+0x39c sctp_iterator_thread(0,ee18cd28,c0fcda70,3d8,c7dde864,...) at sctp_iterator_thread+0x5c fork_exit(c0bc7580,0,ee18cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee18cd60, ebp = 0 --- Tracing command fdc0 pid 4 tid 100065 td 0xc9624bc0 sched_switch(c9624bc0,0,104,1b3,ee17db6c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c9624bc0,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c947163c,5c,c0fbfdd2,0,0,...) at sleepq_timedwait+0x6b _sleep(c947163c,c94716f0,5c,c0fbfdd2,3e8,...) at _sleep+0x36a fdc_thread(c9471600,ee17dd28,c0fcda70,3d8,c7ddeb30,...) at fdc_thread+0x27d fork_exit(c0da3330,c9471600,ee17dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee17dd60, ebp = 0 --- Tracing command fw0_probe pid 3 tid 100063 td 0xc94e32f0 sched_switch(c94e32f0,0,104,1b3,ee16d710,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0a75fdb,...) at mi_switch+0x243 sleepq_switch(c94e32f0,0,c0fd9600,1ac,6c,...) at sleepq_switch+0x1a7 sleepq_catch_signals(c0fd9600,165,0,100,100,...) at sleepq_catch_signals+0xfc sleepq_wait_sig(c952b000,6c,c0fbfdd2,100,0,...) at sleepq_wait_sig+0x17 _sleep(c952b000,c952f488,16c,c0fbfdd2,0,...) at _sleep+0x385 fw_bus_probe_thread(c952b000,ee16dd28,c0fcda70,3d8,c940a000,...) at fw_bus_probe_thread+0xa08 fork_exit(c0701d40,c952b000,ee16dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee16dd60, ebp = 0 --- Tracing command usb pid 15 tid 100059 td 0xc94f6000 sched_switch(c94f6000,0,104,1b3,ee156c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f6000,0,c0fd9600,26e,c94f6000,...) at sleepq_switch+0x1a7 sleepq_wait(c94fdd34,0,ee156cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94fdd34,c94fde4c,c0fb1451,74,c94fdd3c,...) at _cv_wait+0x260 usb_process(c94fdd2c,ee156d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94fdd2c,ee156d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee156d60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc94f62f0 sched_switch(c94f62f0,0,104,1b3,ee153c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f62f0,0,c0fd9600,26e,c94f62f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94fdd04,0,ee153cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94fdd04,c94fde4c,c0fb1451,74,c94fdd0c,...) at _cv_wait+0x260 usb_process(c94fdcfc,ee153d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94fdcfc,ee153d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee153d60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc94f65e0 sched_switch(c94f65e0,0,104,1b3,ee150c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f65e0,0,c0fd9600,26e,c94f65e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94fdcd4,0,ee150cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94fdcd4,c94fde4c,c0fb1451,74,c94fdcdc,...) at _cv_wait+0x260 usb_process(c94fdccc,ee150d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94fdccc,ee150d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee150d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc94f68d0 sched_switch(c94f68d0,0,104,1b3,ee14dc28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f68d0,0,c0fd9600,26e,c94f68d0,...) at sleepq_switch+0x1a7 sleepq_wait(c94fdca4,0,ee14dcac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94fdca4,c94fde4c,c0fb1451,74,c94fdcac,...) at _cv_wait+0x260 usb_process(c94fdc9c,ee14dd28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94fdc9c,ee14dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee14dd60, ebp = 0 --- Tracing command usb pid 15 tid 100054 td 0xc94f8000 sched_switch(c94f8000,0,104,1b3,ee144c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f8000,0,c0fd9600,26e,c94f8000,...) at sleepq_switch+0x1a7 sleepq_wait(c94eeb5c,0,ee144cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94eeb5c,c94eec74,c0fb1451,74,c94eeb64,...) at _cv_wait+0x260 usb_process(c94eeb54,ee144d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94eeb54,ee144d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee144d60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc94f82f0 sched_switch(c94f82f0,0,104,1b3,ee141c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f82f0,0,c0fd9600,26e,c94f82f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94eeb2c,0,ee141cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94eeb2c,c94eec74,c0fb1451,74,c94eeb34,...) at _cv_wait+0x260 usb_process(c94eeb24,ee141d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94eeb24,ee141d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee141d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc94f85e0 sched_switch(c94f85e0,0,104,1b3,ee13ec28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94f85e0,0,c0fd9600,26e,c94f85e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94eeafc,0,ee13ecac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94eeafc,c94eec74,c0fb1451,74,c94eeb04,...) at _cv_wait+0x260 usb_process(c94eeaf4,ee13ed28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94eeaf4,ee13ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee13ed60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc94e1000 sched_switch(c94e1000,0,104,1b3,ee13bc28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e1000,0,c0fd9600,26e,c94e1000,...) at sleepq_switch+0x1a7 sleepq_wait(c94eeacc,0,ee13bcac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94eeacc,c94eec74,c0fb1451,74,c94eead4,...) at _cv_wait+0x260 usb_process(c94eeac4,ee13bd28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94eeac4,ee13bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee13bd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc94e12f0 sched_switch(c94e12f0,0,104,1b3,ee137c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e12f0,0,c0fd9600,26e,c94e12f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e9b5c,0,ee137cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e9b5c,c94e9c74,c0fb1451,74,c94e9b64,...) at _cv_wait+0x260 usb_process(c94e9b54,ee137d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e9b54,ee137d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee137d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc94e15e0 sched_switch(c94e15e0,0,104,1b3,ee134c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e15e0,0,c0fd9600,26e,c94e15e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e9b2c,0,ee134cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e9b2c,c94e9c74,c0fb1451,74,c94e9b34,...) at _cv_wait+0x260 usb_process(c94e9b24,ee134d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e9b24,ee134d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee134d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc94e18d0 sched_switch(c94e18d0,0,104,1b3,ee131c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e18d0,0,c0fd9600,26e,c94e18d0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e9afc,0,ee131cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e9afc,c94e9c74,c0fb1451,74,c94e9b04,...) at _cv_wait+0x260 usb_process(c94e9af4,ee131d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e9af4,ee131d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee131d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc94e1bc0 sched_switch(c94e1bc0,0,104,1b3,ee12ec28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e1bc0,0,c0fd9600,26e,c94e1bc0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e9acc,0,ee12ecac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e9acc,c94e9c74,c0fb1451,74,c94e9ad4,...) at _cv_wait+0x260 usb_process(c94e9ac4,ee12ed28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e9ac4,ee12ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee12ed60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc94e2000 sched_switch(c94e2000,0,104,1b3,ee12ac28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e2000,0,c0fd9600,26e,c94e2000,...) at sleepq_switch+0x1a7 sleepq_wait(c94e4b5c,0,ee12acac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e4b5c,c94e4c74,c0fb1451,74,c94e4b64,...) at _cv_wait+0x260 usb_process(c94e4b54,ee12ad28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e4b54,ee12ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee12ad60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc94e22f0 sched_switch(c94e22f0,0,104,1b3,ee127c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e22f0,0,c0fd9600,26e,c94e22f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e4b2c,0,ee127cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e4b2c,c94e4c74,c0fb1451,74,c94e4b34,...) at _cv_wait+0x260 usb_process(c94e4b24,ee127d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e4b24,ee127d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee127d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc94e25e0 sched_switch(c94e25e0,0,104,1b3,ee124c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e25e0,0,c0fd9600,26e,c94e25e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e4afc,0,ee124cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e4afc,c94e4c74,c0fb1451,74,c94e4b04,...) at _cv_wait+0x260 usb_process(c94e4af4,ee124d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e4af4,ee124d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee124d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc94e28d0 sched_switch(c94e28d0,0,104,1b3,ee121c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e28d0,0,c0fd9600,26e,c94e28d0,...) at sleepq_switch+0x1a7 sleepq_wait(c94e4acc,0,ee121cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94e4acc,c94e4c74,c0fb1451,74,c94e4ad4,...) at _cv_wait+0x260 usb_process(c94e4ac4,ee121d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94e4ac4,ee121d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee121d60, ebp = 0 --- Tracing command usb pid 15 tid 100041 td 0xc94e3000 sched_switch(c94e3000,0,104,1b3,ee11ac28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94e3000,0,c0fd9600,26e,c94e3000,...) at sleepq_switch+0x1a7 sleepq_wait(c94dcb5c,0,ee11acac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94dcb5c,c94dcc74,c0fb1451,74,c94dcb64,...) at _cv_wait+0x260 usb_process(c94dcb54,ee11ad28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94dcb54,ee11ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee11ad60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc94be000 sched_switch(c94be000,0,104,1b3,ee117c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94be000,0,c0fd9600,26e,c94be000,...) at sleepq_switch+0x1a7 sleepq_wait(c94dcb2c,0,ee117cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94dcb2c,c94dcc74,c0fb1451,74,c94dcb34,...) at _cv_wait+0x260 usb_process(c94dcb24,ee117d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94dcb24,ee117d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee117d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc94be2f0 sched_switch(c94be2f0,0,104,1b3,ee114c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94be2f0,0,c0fd9600,26e,c94be2f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94dcafc,0,ee114cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94dcafc,c94dcc74,c0fb1451,74,c94dcb04,...) at _cv_wait+0x260 usb_process(c94dcaf4,ee114d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94dcaf4,ee114d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee114d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc94be5e0 sched_switch(c94be5e0,0,104,1b3,ee111c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94be5e0,0,c0fd9600,26e,c94be5e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94dcacc,0,ee111cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94dcacc,c94dcc74,c0fb1451,74,c94dcad4,...) at _cv_wait+0x260 usb_process(c94dcac4,ee111d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94dcac4,ee111d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee111d60, ebp = 0 --- Tracing command usb pid 15 tid 100036 td 0xc94bebc0 sched_switch(c94bebc0,0,104,1b3,ee10ac28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94bebc0,0,c0fd9600,26e,c94bebc0,...) at sleepq_switch+0x1a7 sleepq_wait(c94d4b5c,0,ee10acac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94d4b5c,c94d4c74,c0fb1451,74,c94d4b64,...) at _cv_wait+0x260 usb_process(c94d4b54,ee10ad28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94d4b54,ee10ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee10ad60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc94c0000 sched_switch(c94c0000,0,104,1b3,ee107c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94c0000,0,c0fd9600,26e,c94c0000,...) at sleepq_switch+0x1a7 sleepq_wait(c94d4b2c,0,ee107cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94d4b2c,c94d4c74,c0fb1451,74,c94d4b34,...) at _cv_wait+0x260 usb_process(c94d4b24,ee107d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94d4b24,ee107d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee107d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc94c02f0 sched_switch(c94c02f0,0,104,1b3,ee104c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94c02f0,0,c0fd9600,26e,c94c02f0,...) at sleepq_switch+0x1a7 sleepq_wait(c94d4afc,0,ee104cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94d4afc,c94d4c74,c0fb1451,74,c94d4b04,...) at _cv_wait+0x260 usb_process(c94d4af4,ee104d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94d4af4,ee104d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee104d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc94c05e0 sched_switch(c94c05e0,0,104,1b3,ee101c28,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c94c05e0,0,c0fd9600,26e,c94c05e0,...) at sleepq_switch+0x1a7 sleepq_wait(c94d4acc,0,ee101cac,1,0,...) at sleepq_wait+0x6b _cv_wait(c94d4acc,c94d4c74,c0fb1451,74,c94d4ad4,...) at _cv_wait+0x260 usb_process(c94d4ac4,ee101d28,c0fcda70,3d8,c940a2cc,...) at usb_process+0x1a3 fork_exit(c092fd00,c94d4ac4,ee101d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee101d60, ebp = 0 --- Tracing command ctl_thrd pid 2 tid 100028 td 0xc7ee98d0 sched_switch(c7ee98d0,0,104,1b3,c7b73674,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbe8c0,...) at mi_switch+0x243 sleepq_switch(c7ee98d0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c7fc4000,5c,c0e38810,0,0,...) at sleepq_wait+0x6b _sleep(c7fc4000,c7fc4000,5c,c0e38810,0,...) at _sleep+0x39c ctl_work_thread(c7fc4000,c7b73d28,c0fcda70,3d8,c940a598,...) at ctl_work_thread+0x22f6 fork_exit(c04d0d70,c7fc4000,c7b73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b73d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc7ee85e0 sched_switch(c7ee85e0,0,104,1b3,c7b51bf0,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c7ee85e0,0,c0fd9600,291,2,...) at sleepq_switch+0x1a7 sleepq_timedwait(c118f5e4,0,c0fbfdd2,2,0,...) at sleepq_timedwait+0x6b _sleep(c118f5e4,0,0,c0fbfdd2,64) at _sleep+0x36a pause(c0fbfdd2,64,c0f8d4b0,111,0,...) at pause+0xa0 random_kthread(0,c7b51d28,c0fcda70,3d8,c7ddd000,...) at random_kthread+0x1ef fork_exit(c082d560,0,c7b51d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b51d60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc7ee8bc0 sched_switch(c7ee8bc0,0,104,1b3,c7b4bbd4,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbe9c0,...) at mi_switch+0x243 sleepq_switch(c7ee8bc0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c118c6a8,5c,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c118c6a8,c118c608,25c,c0fbfdd2,0,...) at _sleep+0x39c g_io_schedule_down(c7ee8bc0,0,c0fc1921,6c,c7b4bd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c7b4bd28,c0fcda70,3d8,c7ddd2cc,...) at g_down_procbody+0x9d fork_exit(c09e7200,0,c7b4bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b4bd60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc7ee9000 sched_switch(c7ee9000,0,104,1b3,c7b48bfc,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbe980,...) at mi_switch+0x243 sleepq_switch(c7ee9000,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c118c6a4,5c,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c118c6a4,c118c628,25c,c0fbfdd2,0,...) at _sleep+0x39c g_io_schedule_up(c7ee9000,0,c0fc1921,5f,c7b48d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c7b48d28,c0fcda70,3d8,c7ddd2cc,...) at g_up_procbody+0x9d fork_exit(c09e72a0,0,c7b48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b48d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc7de02f0 sched_switch(c7de02f0,0,104,1b3,c7b45bf8,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbf680,...) at mi_switch+0x243 sleepq_switch(c7de02f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c118c69c,5c,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c118c69c,c118c5cc,25c,c0fbfdd2,0,...) at _sleep+0x39c g_run_events(c119d440,0,c0fc1921,79,c7b45d14,...) at g_run_events+0x54b g_event_procbody(0,c7b45d28,c0fcda70,3d8,c7ddd2cc,...) at g_event_procbody+0x90 fork_exit(c09e7340,0,c7b45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b45d60, ebp = 0 --- Tracing command intr pid 12 tid 100067 td 0xc96245e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100066 td 0xc96248d0 sched_switch(c96248d0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c96248d0,...) at mi_switch+0x243 ithread_loop(c961d990,ee180d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c961d990,ee180d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee180d60, ebp = 0 --- Tracing command intr pid 12 tid 100064 td 0xc9625000 sched_switch(c9625000,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c9625000,...) at mi_switch+0x243 ithread_loop(c9617a40,ee170d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9617a40,ee170d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee170d60, ebp = 0 --- Tracing command intr pid 12 tid 100061 td 0xc94e38d0 sched_switch(c94e38d0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94e38d0,...) at mi_switch+0x243 ithread_loop(c9514880,ee164d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9514880,ee164d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee164d60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc94e3bc0 sched_switch(c94e3bc0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94e3bc0,...) at mi_switch+0x243 ithread_loop(c9514010,ee15cd28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9514010,ee15cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee15cd60, ebp = 0 --- Tracing command intr pid 12 tid 100055 td 0xc94f6bc0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100042 td 0xc94e2bc0 sched_switch(c94e2bc0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94e2bc0,...) at mi_switch+0x243 ithread_loop(c94c4720,ee11ed28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c94c4720,ee11ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee11ed60, ebp = 0 --- Tracing command intr pid 12 tid 100037 td 0xc94be8d0 sched_switch(c94be8d0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94be8d0,...) at mi_switch+0x243 ithread_loop(c94ac590,ee10ed28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c94ac590,ee10ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee10ed60, ebp = 0 --- Tracing command intr pid 12 tid 100032 td 0xc94c08d0 sched_switch(c94c08d0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94c08d0,...) at mi_switch+0x243 ithread_loop(c94ac830,ee0fed28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c94ac830,ee0fed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee0fed60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc94c0bc0 sched_switch(c94c0bc0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c94c0bc0,...) at mi_switch+0x243 ithread_loop(c9496ae0,edfa8d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9496ae0,edfa8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xedfa8d60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc7ee92f0 sched_switch(c7ee92f0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7ee92f0,...) at mi_switch+0x243 ithread_loop(c9496c80,edfa4d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9496c80,edfa4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xedfa4d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc7ee95e0 sched_switch(c7ee95e0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7ee95e0,...) at mi_switch+0x243 ithread_loop(c9496500,c7ba4d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c9496500,c7ba4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7ba4d60, ebp = 0 --- Tracing command intr pid 12 tid 100025 td 0xc7fc12f0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xc7e22bc0 sched_switch(c7e22bc0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7e22bc0,...) at mi_switch+0x243 ithread_loop(c7db3470,c7b5bd28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db3470,c7b5bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b5bd60, ebp = 0 --- Tracing command intr pid 12 tid 100019 td 0xc7ee8000 sched_switch(c7ee8000,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7ee8000,...) at mi_switch+0x243 ithread_loop(c7db34b0,c7b58d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db34b0,c7b58d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b58d60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc7ee82f0 sched_switch(c7ee82f0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7ee82f0,...) at mi_switch+0x243 ithread_loop(c7db34c0,c7b55d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db34c0,c7b55d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b55d60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc7de05e0 sched_switch(c7de05e0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7de05e0,...) at mi_switch+0x243 ithread_loop(c7db36f0,c7b42d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db36f0,c7b42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b42d60, ebp = 0 --- Tracing command intr pid 12 tid 100011 td 0xc7de08d0 sched_switch(c7de08d0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7de08d0,...) at mi_switch+0x243 ithread_loop(c7db3700,c7b3fd28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db3700,c7b3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b3fd60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc7de0bc0 sched_switch(c7de0bc0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7de0bc0,...) at mi_switch+0x243 ithread_loop(c7db3710,c7b3cd28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db3710,c7b3cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b3cd60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc7e22000 sched_switch(c7e22000,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7e22000,...) at mi_switch+0x243 ithread_loop(c7db3720,c7b39d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db3720,c7b39d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b39d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc7e222f0 sched_switch(c7e222f0,0,109,1b3,0,...) at sched_switch+0x453 mi_switch(109,0,c0fcdd93,557,c7e222f0,...) at mi_switch+0x243 ithread_loop(c7db3730,c7b36d28,c0fcda70,3d8,c7ddd598,...) at ithread_loop+0x1fb fork_exit(c0a5b140,c7db3730,c7b36d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b36d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc7e225e0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xc7ddf000 cpustop_handler(3,c7b2fbfc,c0e01f73,c7b1cc80,c7b2fb4c,...) at cpustop_handler+0x26 ipi_nmi_handler(c7b1cc80,c7b2fb4c,c0a74a53,c7b1cc80,c7ddd864,...) at ipi_nmi_handler+0x35 trap(c7b2fc08) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0de7765, esp = 0xc7b2fc48, ebp = 0xc7b2fc48 --- acpi_cpu_c1(c7b2fc64,c1153730,3,317,11f44,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b2fc98,c0df49c9,0,3,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,3,0,2,c7b2fce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b2fcc0,c0fd5c2c,3e5,c7ddf000,...) at cpu_idle+0x89 sched_idletd(0,c7b2fd28,c0fcda70,3d8,c7ddd864,...) at sched_idletd+0x235 fork_exit(c0ab0170,0,c7b2fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b2fd60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc7ddf2f0 kdb_enter(c0e47e69,c0fd7bbe,c7b2cb9c,c08fb9bd,c7da7200,...) at kdb_enter+0x3a kdb_break(c7da7200,c0e1cb9e,1,c119e140,c7b2cbd0,...) at kdb_break+0x28 uart_intr(c7da7200,c133d2f8,c7b2cbdc,c0e1cb9e,0,...) at uart_intr+0x12d intr_event_handle(c7daab80,c7b2cc08,c136f580,0,c9470800,...) at intr_event_handle+0x76 intr_execute_handlers(c7e210d0,c7b2cc08,2,c7b2cc48,c0dec1f5,...) at intr_execute_handlers+0x49 lapic_handle_intr(30,c7b2cc08) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0de7765, esp = 0xc7b2cc48, ebp = 0xc7b2cc48 --- acpi_cpu_c1(c7b2cc64,c1153730,2,317,11f44,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b2cc98,c0df49c9,0,2,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,2,0,2,c7b2cce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b2ccc0,c0fd5c2c,a41,c7ddf2f0,...) at cpu_idle+0x89 sched_idletd(0,c7b2cd28,c0fcda70,3d8,c7ddd864,...) at sched_idletd+0x235 fork_exit(c0ab0170,0,c7b2cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b2cd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc7ddf5e0 cpustop_handler(1,c7b29bfc,c0e01f73,c7b16c80,c7b29b4c,...) at cpustop_handler+0x26 ipi_nmi_handler(c7b16c80,c7b29b4c,c0a74a53,c7b16c80,c7ddd864,...) at ipi_nmi_handler+0x35 trap(c7b29c08) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0de7765, esp = 0xc7b29c48, ebp = 0xc7b29c48 --- acpi_cpu_c1(c7b29c64,c1153730,1,317,11f44,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b29c98,c0df49c9,0,1,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,1,0,2,c7b29ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b29cc0,c0fd5c2c,3e5,c7ddf5e0,...) at cpu_idle+0x89 sched_idletd(0,c7b29d28,c0fcda70,3d8,c7ddd864,...) at sched_idletd+0x235 fork_exit(c0ab0170,0,c7b29d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b29d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc7ddf8d0 cpustop_handler(0,c7b26bfc,c0e01f73,c1825c80,c7b26b4c,...) at cpustop_handler+0x26 ipi_nmi_handler(c1825c80,c7b26b4c,c0a74a53,c1825c80,c7ddd864,...) at ipi_nmi_handler+0x35 trap(c7b26c08) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0de7765, esp = 0xc7b26c48, ebp = 0xc7b26c48 --- acpi_cpu_c1(c7b26c64,c1153730,0,317,11f44,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c7b26c98,c0df49c9,0,0,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,0,0,2,c7b26ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c7b26cc0,c0fd5c2c,a41,c7ddf8d0,...) at cpu_idle+0x89 sched_idletd(0,c7b26d28,c0fcda70,3d8,c7ddd864,...) at sched_idletd+0x235 fork_exit(c0ab0170,0,c7b26d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b26d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc7ddfbc0 sched_switch(c7ddfbc0,0,104,1b3,c7b22acc,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0a75fdb,...) at mi_switch+0x243 sleepq_switch(c7ddfbc0,0,c0fd9600,1ac,6c,...) at sleepq_switch+0x1a7 sleepq_catch_signals(c0fd9600,165,0,100,100,...) at sleepq_catch_signals+0xfc sleepq_wait_sig(c7dddb30,6c,c0fdc538,100,0,...) at sleepq_wait_sig+0x17 _sleep(c7dddb30,c7dddbb8,16c,c0fdc538,0,...) at _sleep+0x385 kern_wait(c7ddfbc0,ffffffff,c7b22c58,2,0,...) at kern_wait+0x597 sys_wait4(c7ddfbc0,c7b22cec,c102715c,c0fcdafc,e,...) at sys_wait4+0x3b syscall(c7b22d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x806070f, esp = 0xbfbfe80c, ebp = 0xbfbfe828 --- Tracing command audit pid 10 tid 100001 td 0xc7de0000 sched_switch(c7de0000,0,104,1b3,c7b1fc08,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,eb,...) at mi_switch+0x243 sleepq_switch(c7de0000,0,c0fd9600,26e,c7de0000,...) at sleepq_switch+0x1a7 sleepq_wait(c130de00,0,c7b1fc8c,1,0,...) at sleepq_wait+0x6b _cv_wait(c130de00,c130dde4,c1005917,194,c7b1fcc4,...) at _cv_wait+0x260 audit_worker(0,c7b1fd28,c0fcda70,3d8,c7dde000,...) at audit_worker+0x84 fork_exit(c0c971c0,0,c7b1fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b1fd60, ebp = 0 --- Tracing command kernel pid 0 tid 5171773 td 0xcc3f12f0 sched_switch(cc3f12f0,0,104,1b3,f33e4c1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cb758340,...) at mi_switch+0x243 sleepq_switch(cc3f12f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(cceac840,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(cceac840,cceac858,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(ca73a620,f33e4d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,ca73a620,f33e4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf33e4d60, ebp = 0 --- Tracing command kernel pid 0 tid 5171772 td 0xcc12d8d0 sched_switch(cc12d8d0,0,104,1b3,f33e1c1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cb758340,...) at mi_switch+0x243 sleepq_switch(cc12d8d0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(cceac840,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(cceac840,cceac858,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(ca73a620,f33e1d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,ca73a620,f33e1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf33e1d60, ebp = 0 --- Tracing command kernel pid 0 tid 5171771 td 0xcc17e5e0 sched_switch(cc17e5e0,0,104,1b3,f33dec1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cb758340,...) at mi_switch+0x243 sleepq_switch(cc17e5e0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(cceac840,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(cceac840,cceac858,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(ca73a620,f33ded28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,ca73a620,f33ded28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf33ded60, ebp = 0 --- Tracing command kernel pid 0 tid 5171770 td 0xcc1712f0 sched_switch(cc1712f0,0,104,1b3,f33dbc1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,cb758340,...) at mi_switch+0x243 sleepq_switch(cc1712f0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(cceac840,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(cceac840,cceac858,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(ca73a620,f33dbd28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,ca73a620,f33dbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf33dbd60, ebp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xc9624000 sched_switch(c9624000,0,104,1b3,ee189be8,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c9624000,0,c0fd9600,291,2,...) at sleepq_switch+0x1a7 sleepq_timedwait(c118f5e4,0,c0fbfdd2,2,0,...) at sleepq_timedwait+0x6b _sleep(c118f5e4,0,0,c0fbfdd2,bb8) at _sleep+0x36a pause(c0fbfdd2,bb8,12b,129,c118c8e8,...) at pause+0xa0 deadlkres(0,ee189d28,c0fcda70,3d8,c118c860,...) at deadlkres+0x324 fork_exit(c0a39fc0,0,ee189d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee189d60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xc96242f0 sched_switch(c96242f0,0,104,1b3,ee186c38,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd9600,...) at mi_switch+0x243 sleepq_switch(c96242f0,0,c0fd9600,26e,c96242f0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0bdc0,0,c0fd4721,c0fbfdd2,0,...) at sleepq_wait+0x6b msleep_spin(c7f0bdc0,c7f0bdd8,c0fbfdd2,0,a77,...) at msleep_spin+0x25f taskqueue_thread_loop(c136f640,ee186d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0x8f fork_exit(c0acd400,c136f640,ee186d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee186d60, ebp = 0 --- Tracing command kernel pid 0 tid 100062 td 0xc94e35e0 sched_switch(c94e35e0,0,104,1b3,ee16ac38,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd9600,...) at mi_switch+0x243 sleepq_switch(c94e35e0,0,c0fd9600,26e,c94e35e0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0be00,0,c0fd4721,c0fbfdd2,0,...) at sleepq_wait+0x6b msleep_spin(c7f0be00,c7f0be18,c0fbfdd2,0,a77,...) at msleep_spin+0x25f taskqueue_thread_loop(c952f49c,ee16ad28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0x8f fork_exit(c0acd400,c952f49c,ee16ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee16ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc7ee9bc0 sched_switch(c7ee9bc0,0,104,1b3,c7b70c1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c94f7200,...) at mi_switch+0x243 sleepq_switch(c7ee9bc0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0be40,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c7f0be40,c7f0be58,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(c11ac1ac,c7b70d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,c11ac1ac,c7b70d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b70d60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc7fc1000 sched_switch(c7fc1000,0,104,1b3,c7b6dc1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,ca7db600,...) at mi_switch+0x243 sleepq_switch(c7fc1000,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0be80,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c7f0be80,c7f0be98,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(c130e3e4,c7b6dd28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,c130e3e4,c7b6dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b6dd60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc7fc15e0 sched_switch(c7fc15e0,0,104,1b3,c7b67c1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbe7c0,...) at mi_switch+0x243 sleepq_switch(c7fc15e0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0c040,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c7f0c040,c7f0c058,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(c118d1f8,c7b67d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,c118d1f8,c7b67d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b67d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc7fc18d0 sched_switch(c7fc18d0,0,104,1b3,c7b64c38,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd9600,...) at mi_switch+0x243 sleepq_switch(c7fc18d0,0,c0fd9600,26e,c7fc18d0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0c080,0,c0fd4721,c0fbfdd2,0,...) at sleepq_wait+0x6b msleep_spin(c7f0c080,c7f0c098,c0fbfdd2,0,a77,...) at msleep_spin+0x25f taskqueue_thread_loop(c1156398,c7b64d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0x8f fork_exit(c0acd400,c1156398,c7b64d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b64d60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc7fc1bc0 sched_switch(c7fc1bc0,0,104,1b3,c7b61c38,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd9600,...) at mi_switch+0x243 sleepq_switch(c7fc1bc0,0,c0fd9600,26e,c7fc1bc0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0c080,0,c0fd4721,c0fbfdd2,0,...) at sleepq_wait+0x6b msleep_spin(c7f0c080,c7f0c098,c0fbfdd2,0,a77,...) at msleep_spin+0x25f taskqueue_thread_loop(c1156398,c7b61d28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0x8f fork_exit(c0acd400,c1156398,c7b61d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b61d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc7e228d0 sched_switch(c7e228d0,0,104,1b3,c7b5ec38,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c0fd9600,...) at mi_switch+0x243 sleepq_switch(c7e228d0,0,c0fd9600,26e,c7e228d0,...) at sleepq_switch+0x1a7 sleepq_wait(c7f0c080,0,c0fd4721,c0fbfdd2,0,...) at sleepq_wait+0x6b msleep_spin(c7f0c080,c7f0c098,c0fbfdd2,0,a77,...) at msleep_spin+0x25f taskqueue_thread_loop(c1156398,c7b5ed28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0x8f fork_exit(c0acd400,c1156398,c7b5ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b5ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc7ee88d0 sched_switch(c7ee88d0,0,104,1b3,c7b4ec1c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,c7dbea00,...) at mi_switch+0x243 sleepq_switch(c7ee88d0,0,c0fd9600,26e,0,...) at sleepq_switch+0x1a7 sleepq_wait(c7ef0000,0,c0fbfdd2,0,0,...) at sleepq_wait+0x6b _sleep(c7ef0000,c7ef0018,0,c0fbfdd2,0,...) at _sleep+0x39c taskqueue_thread_loop(c11aabc0,c7b4ed28,c0fcda70,3d8,c118c860,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acd400,c11aabc0,c7b4ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc7b4ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc118cb30 sched_switch(c118cb30,0,104,1b3,c1820c7c,...) at sched_switch+0x453 mi_switch(104,0,c0fd9600,1f8,352,...) at mi_switch+0x243 sleepq_switch(c118cb30,0,c0fd9600,291,0,...) at sleepq_switch+0x1a7 sleepq_timedwait(c118c860,54,c10062f8,0,0,...) at sleepq_timedwait+0x6b _sleep(c118c860,0,54,c10062f8,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x243 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Dumping 476 MB: 461 445 429 413 397 381 365 349 333 317 301 285 269 253 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) proc 62311 [Switching to thread 88 (Thread 5171748)]#0 sched_switch (td=0xcc1a62f0, newtd=0xc7ddf8d0, flags=0x104) at ../../../kern/sched_ule.c:1927 1927 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xcc1a62f0, newtd=0xc7ddf8d0, flags=0x104) at ../../../kern/sched_ule.c:1927 #1 0xc0a915b3 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:485 #2 0xc0aca2c7 in sleepq_switch (wchan=0xd203f5f8, pri=0x60) at ../../../kern/subr_sleepqueue.c:544 #3 0xc0acb00b in sleepq_wait (wchan=0xd203f5f8, pri=0x60) at ../../../kern/subr_sleepqueue.c:623 #4 0xc0a6ccc6 in __lockmgr_args (lk=0xd203f5f8, flags=0x200501, ilk=0xd203f664, wmesg=0x0, pri=0x0, timo=0x0, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at ../../../kern/kern_lock.c:225 #5 0xc0b15e25 in vop_stdlock (ap=0xf3399590) at lockmgr.h:97 #6 0xc0e28853 in VOP_LOCK1_APV (vop=0xc11159e0, a=0xf3399590) at vnode_if.c:1988 #7 0xc0b39748 in _vn_lock (vp=0xd203f5a0, flags=0x200501, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at vnode_if.h:859 #8 0xc0b2bcbb in vget (vp=0xd203f5a0, flags=0x200401, td=0xcc1a62f0) at ../../../kern/vfs_subr.c:2160 #9 0xc0d0503d in vm_fault_hold (map=0xcad371d0, vaddr=0x281b7000, fault_type=0x1, fault_flags=Variable "fault_flags" is not available. ) at ../../../vm/vm_fault.c:549 #10 0xc0d066ab in vm_fault (map=0xcad371d0, vaddr=0x281b7000, fault_type=Variable "fault_type" is not available. ) at ../../../vm/vm_fault.c:229 #11 0xc0e01519 in trap_pfault (frame=0xf3399920, usermode=0x0, eva=0x281b7000) at ../../../i386/i386/trap.c:903 #12 0xc0e02531 in trap (frame=0xf3399920) at ../../../i386/i386/trap.c:546 #13 0xc0debdfc in calltrap () at ../../../i386/i386/exception.s:169 #14 0xc0dff312 in copyin () at ../../../i386/i386/support.s:362 #15 0xf3399c5c in ?? () #16 0x00000001 in ?? () #17 0x00009000 in ?? () #18 0x00000000 in ?? () #19 0x00001000 in ?? () #20 0x00000000 in ?? () #21 0x00000001 in ?? () #22 0xcc1a62f0 in ?? () #23 0x281b7000 in ?? () #24 0x00001000 in ?? () #25 0xf3399d1c in ?? () #26 0xc0e01a2e in syscall (frame=0xf33999fc) at subr_syscall.c:135 Previous frame inner to this frame (corrupt stack?) (kgdb) f 7 #7 0xc0b39748 in _vn_lock (vp=0xd203f5a0, flags=0x200501, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at vnode_if.h:859 859 return (VOP_LOCK1_APV(vp->v_op, &a)); (kgdb) p *vp $1 = {v_type = VREG, v_tag = 0xd48d7014 "tmpfs", v_op = 0xd48d80a0, v_data = 0xcccd7110, v_mount = 0xcaa842dc, v_nmntvnodes = {tqe_next = 0xd2b089d8, tqe_prev = 0xd070844c}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0x0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xd203f5d0}, v_cache_dd = 0x0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xd48d7014 "tmpfs", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7d8a2a8}, lk_lock = 0xcb7db5e4, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xb, pcs = {0xc0a6cdae, 0xc0b15e25, 0xc0e28853, 0xc0b39748, 0xc0b3a436, 0xc0b3809a, 0xc0ad7de5, 0xc0ad8458, 0xc0ad856f, 0xc0e01a2e, 0xc0debe91, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = {lo_name = 0xc0fdb8bc "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d821e0}, mtx_lock = 0x4}, v_vnlock = 0xd203f5f8, v_holdcnt = 0x9, v_usecount = 0x5, v_iflag = 0x200, v_vflag = 0x0, v_writecount = 0x2, v_actfreelist = {tqe_next = 0xce30e870, tqe_prev = 0xd2b08ac8}, v_bufobj = {bo_mtx = {lock_object = { lo_name = 0xc0fe474a "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d87ba8}, mtx_lock = 0x4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd203f6ac}, bv_root = 0x0, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd203f6bc}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xc1113ca0, bo_bsize = 0x1000, bo_object = 0xce3c58ac, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xd203f5a0, __bo_vnode = 0xd203f5a0}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xd203f6fc}, rl_currdep = 0x0}} (kgdb) (kgdb) f 7 #7 0xc0b39748 in _vn_lock (vp=0xd203f5a0, flags=0x200501, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at vnode_if.h:859 859 return (VOP_LOCK1_APV(vp->v_op, &a)); (kgdb) p *vp $1 = {v_type = VREG, v_tag = 0xd48d7014 "tmpfs", v_op = 0xd48d80a0, v_data = 0xcccd7110, v_mount = 0xcaa842dc, v_nmntvnodes = {tqe_next = 0xd2b089d8, tqe_prev = 0xd070844c}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0x0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xd203f5d0}, v_cache_dd = 0x0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xd48d7014 "tmpfs", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7d8a2a8}, lk_lock = 0xcb7db5e4, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xb, pcs = {0xc0a6cdae, 0xc0b15e25, 0xc0e28853, 0xc0b39748, 0xc0b3a436, 0xc0b3809a, 0xc0ad7de5, 0xc0ad8458, 0xc0ad856f, 0xc0e01a2e, 0xc0debe91, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = {lo_name = 0xc0fdb8bc "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d821e0}, mtx_lock = 0x4}, v_vnlock = 0xd203f5f8, v_holdcnt = 0x9, v_usecount = 0x5, v_iflag = 0x200, v_vflag = 0x0, v_writecount = 0x2, v_actfreelist = {tqe_next = 0xce30e870, tqe_prev = 0xd2b08ac8}, v_bufobj = {bo_mtx = {lock_object = { lo_name = 0xc0fe474a "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d87ba8}, mtx_lock = 0x4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd203f6ac}, bv_root = 0x0, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd203f6bc}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xc1113ca0, bo_bsize = 0x1000, bo_object = 0xce3c58ac, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xd203f5a0, __bo_vnode = 0xd203f5a0}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xd203f6fc}, rl_currdep = 0x0}} (kgdb) p *vp->vn_lock There is no member named vn_lock. (kgdb) p *vp->v_vnlock $2 = {lock_object = {lo_name = 0xd48d7014 "tmpfs", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7d8a2a8}, lk_lock = 0xcb7db5e4, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xb, pcs = { 0xc0a6cdae, 0xc0b15e25, 0xc0e28853, 0xc0b39748, 0xc0b3a436, 0xc0b3809a, 0xc0ad7de5, 0xc0ad8458, 0xc0ad856f, 0xc0e01a2e, 0xc0debe91, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}} (kgdb) (kgdb) proc 62312 [Switching to thread 89 (Thread 100610)]#0 sched_switch (td=0xcb7db5e0, newtd=0xc7ddf000, flags=0x104) at ../../../kern/sched_ule.c:1927 1927 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xcb7db5e0, newtd=0xc7ddf000, flags=0x104) at ../../../kern/sched_ule.c:1927 #1 0xc0a915b3 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:485 #2 0xc0aca2c7 in sleepq_switch (wchan=0xd2b08a30, pri=0x60) at ../../../kern/subr_sleepqueue.c:544 #3 0xc0acb00b in sleepq_wait (wchan=0xd2b08a30, pri=0x60) at ../../../kern/subr_sleepqueue.c:623 #4 0xc0a6ccc6 in __lockmgr_args (lk=0xd2b08a30, flags=0x200501, ilk=0xd2b08a9c, wmesg=0x0, pri=0x0, timo=0x0, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at ../../../kern/kern_lock.c:225 #5 0xc0b15e25 in vop_stdlock (ap=0xf2f60590) at lockmgr.h:97 #6 0xc0e28853 in VOP_LOCK1_APV (vop=0xc11159e0, a=0xf2f60590) at vnode_if.c:1988 #7 0xc0b39748 in _vn_lock (vp=0xd2b089d8, flags=0x200501, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at vnode_if.h:859 #8 0xc0b2bcbb in vget (vp=0xd2b089d8, flags=0x200401, td=0xcb7db5e0) at ../../../kern/vfs_subr.c:2160 #9 0xc0d0503d in vm_fault_hold (map=0xcaeda9f8, vaddr=0x281b5000, fault_type=0x1, fault_flags=Variable "fault_flags" is not available. ) at ../../../vm/vm_fault.c:549 #10 0xc0d066ab in vm_fault (map=0xcaeda9f8, vaddr=0x281b5000, fault_type=Variable "fault_type" is not available. ) at ../../../vm/vm_fault.c:229 #11 0xc0e01519 in trap_pfault (frame=0xf2f60920, usermode=0x0, eva=0x281b5000) at ../../../i386/i386/trap.c:903 #12 0xc0e02531 in trap (frame=0xf2f60920) at ../../../i386/i386/trap.c:546 #13 0xc0debdfc in calltrap () at ../../../i386/i386/exception.s:169 #14 0xc0dff312 in copyin () at ../../../i386/i386/support.s:362 #15 0xf2f60c5c in ?? () #16 0x00000001 in ?? () #17 0x00009000 in ?? () #18 0x00000000 in ?? () #19 0x00001000 in ?? () #20 0x00000000 in ?? () #21 0x00000001 in ?? () #22 0xcb7db5e0 in ?? () #23 0x281b5000 in ?? () #24 0x00001000 in ?? () #25 0xf2f60d1c in ?? () #26 0xc0e01a2e in syscall (frame=0xf2f609fc) at subr_syscall.c:135 Previous frame inner to this frame (corrupt stack?) (kgdb) f 7 #7 0xc0b39748 in _vn_lock (vp=0xd2b089d8, flags=0x200501, file=0xc0fe3751 "../../../kern/vfs_subr.c", line=0x870) at vnode_if.h:859 859 return (VOP_LOCK1_APV(vp->v_op, &a)); (kgdb) p *vp $5 = {v_type = VREG, v_tag = 0xd48d7014 "tmpfs", v_op = 0xd48d80a0, v_data = 0xcccd7088, v_mount = 0xcaa842dc, v_nmntvnodes = {tqe_next = 0xd1cb9438, tqe_prev = 0xd203f5b4}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0x0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xd2b08a08}, v_cache_dd = 0x0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xd48d7014 "tmpfs", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7d8a2a8}, lk_lock = 0xcc1a62f4, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xb, pcs = {0xc0a6cdae, 0xc0b15e25, 0xc0e28853, 0xc0b39748, 0xc0b3a436, 0xc0b3809a, 0xc0ad7de5, 0xc0ad8458, 0xc0ad856f, 0xc0e01a2e, 0xc0debe91, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = {lo_name = 0xc0fdb8bc "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d821e0}, mtx_lock = 0x4}, v_vnlock = 0xd2b08a30, v_holdcnt = 0x6, v_usecount = 0x3, v_iflag = 0x200, v_vflag = 0x0, v_writecount = 0x2, v_actfreelist = {tqe_next = 0xd203f5a0, tqe_prev = 0xd1cb9528}, v_bufobj = {bo_mtx = {lock_object = { lo_name = 0xc0fe474a "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d87ba8}, mtx_lock = 0x4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd2b08ae4}, bv_root = 0x0, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xd2b08af4}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xc1113ca0, bo_bsize = 0x1000, bo_object = 0xd1f1c1bc, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xd2b089d8, __bo_vnode = 0xd2b089d8}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xd2b08b34}, rl_currdep = 0x0}} (kgdb) p *vp->v_mount $6 = {mnt_mtx = {lock_object = {lo_name = 0xc0fbf591 "struct mount mtx", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc7d82450}, mtx_lock = 0x4}, mnt_gen = 0x169, mnt_list = {tqe_next = 0x0, tqe_prev = 0xd00e2018}, mnt_op = 0xd48d8340, mnt_vfc = 0xd48d8440, mnt_vnodecovered = 0xcd9e3438, mnt_syncer = 0xd0708438, mnt_ref = 0x8, mnt_nvnodelist = {tqh_first = 0xce30e870, tqh_last = 0xce4999ec}, mnt_nvnodelistsize = 0x6, mnt_activevnodelist = {tqh_first = 0xce4999d8, tqh_last = 0xd0708528}, mnt_activevnodelistsize = 0x6, mnt_writeopcount = 0x2, mnt_kern_flag = 0x20000000, mnt_flag = 0x1100, mnt_opt = 0xca778a70, mnt_optnew = 0x0, mnt_maxsymlinklen = 0x0, mnt_stat = {f_version = 0x20030518, f_type = 0x87, f_flags = 0x1100, f_bsize = 0x1000, f_iosize = 0x1000, f_blocks = 0x48a440, f_bfree = 0x48a41b, f_bavail = 0x48a41b, f_files = 0x7fffffff, f_ffree = 0x7ffffffa, f_syncwrites = 0x0, f_asyncwrites = 0x0, f_syncreads = 0x0, f_asyncreads = 0x0, f_spare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, f_namemax = 0xff, f_owner = 0x0, f_fsid = {val = {0x8741ff84, 0x87}}, f_charspare = '\0' , f_fstypename = "tmpfs\000\000\000\000\000\000\000\000\000\000", f_mntfromname = "tmpfs", '\0' , f_mntonname = "/mnt", '\0' }, mnt_cred = 0xcaa89a00, mnt_data = 0xcaef0700, mnt_time = 0x0, mnt_iosize_max = 0x10000, mnt_export = 0xcee05b00, mnt_label = 0x0, mnt_hashseed = 0xf908c3d, mnt_lockref = 0x0, mnt_secondary_writes = 0x0, mnt_secondary_accwrites = 0x0, mnt_susp_owner = 0x0, mnt_gjprovider = 0x0, mnt_explock = {lock_object = { lo_name = 0xc0fbf5a2 "explock", lo_flags = 0x5730000, lo_data = 0x0, lo_witness = 0xc7d87ee8}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x6, pcs = {0xc0a6cdae, 0xc0b17717, 0xc0b202a3, 0xc0b209bf, 0xc0e01a2e, 0xc0debe91, 0x0 }}}} (kgdb) p *vp->v_vnlock $7 = {lock_object = {lo_name = 0xd48d7014 "tmpfs", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7d8a2a8}, lk_lock = 0xcc1a62f4, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0xb, pcs = { 0xc0a6cdae, 0xc0b15e25, 0xc0e28853, 0xc0b39748, 0xc0b3a436, 0xc0b3809a, 0xc0ad7de5, 0xc0ad8458, 0xc0ad856f, 0xc0e01a2e, 0xc0debe91, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/fs/nfsclient/nfs_clbio.c =================================================================== --- /usr/src/sys/fs/nfsclient/nfs_clbio.c (revision 235546) +++ /usr/src/sys/fs/nfsclient/nfs_clbio.c (working copy) @@ -722,7 +722,7 @@ ncl_bioread(struct vnode *vp, struct uio *uio, int }; if (n > 0) { - error = uiomove(bp->b_data + on, (int)n, uio); + error = vn_io_fault_uiomove(bp->b_data + on, (int)n, uio); } if (vp->v_type == VLNK) n = 0; @@ -897,8 +897,9 @@ ncl_write(struct vop_write_args *ap) struct nfsmount *nmp = VFSTONFS(vp->v_mount); daddr_t lbn; int bcount; - int n, on, error = 0; - off_t tmp_off; + int bp_cached, n, on, error = 0; + size_t orig_resid, local_resid; + off_t orig_size, tmp_off; KASSERT(uio->uio_rw == UIO_WRITE, ("ncl_write mode")); KASSERT(uio->uio_segflg != UIO_USERSPACE || uio->uio_td == curthread, @@ -950,6 +951,11 @@ flush_and_restart: mtx_unlock(&np->n_mtx); } + orig_resid = uio->uio_resid; + mtx_lock(&np->n_mtx); + orig_size = np->n_size; + mtx_unlock(&np->n_mtx); + /* * If IO_APPEND then load uio_offset. We restart here if we cannot * get the append lock. @@ -1127,7 +1133,10 @@ again: * normally. */ + bp_cached = 1; if (on == 0 && n == bcount) { + if ((bp->b_flags & B_CACHE) == 0) + bp_cached = 0; bp->b_flags |= B_CACHE; bp->b_flags &= ~B_INVAL; bp->b_ioflags &= ~BIO_ERROR; @@ -1178,7 +1187,7 @@ again: * significant cache coherency problems with multiple clients, * especially if locking is implemented later on. * - * as an optimization we could theoretically maintain + * As an optimization we could theoretically maintain * a linked list of discontinuous areas, but we would still * have to commit them separately so there isn't much * advantage to it except perhaps a bit of asynchronization. @@ -1193,8 +1202,24 @@ again: goto again; } - error = uiomove((char *)bp->b_data + on, n, uio); + local_resid = uio->uio_resid; + error = vn_io_fault_uiomove((char *)bp->b_data + on, n, uio); + if (error != 0 && !bp_cached) { + /* + * This block has no other content then what + * possibly was written by faulty uiomove. + * Release it, forgetting the data pages, to + * prevent the leak of uninitialized data to + * usermode. + */ + bp->b_ioflags |= BIO_ERROR; + brelse(bp); + uio->uio_offset -= local_resid - uio->uio_resid; + uio->uio_resid = local_resid; + break; + } + /* * Since this block is being modified, it must be written * again and not just committed. Since write clustering does @@ -1203,17 +1228,18 @@ again: */ bp->b_flags &= ~(B_NEEDCOMMIT | B_CLUSTEROK); - if (error) { - bp->b_ioflags |= BIO_ERROR; - brelse(bp); - break; - } + /* + * Get the partial update on the progress made from + * uiomove, if error occured. + */ + if (error != 0) + n = local_resid - uio->uio_resid; /* * Only update dirtyoff/dirtyend if not a degenerate * condition. */ - if (n) { + if (n > 0) { if (bp->b_dirtyend > 0) { bp->b_dirtyoff = min(on, bp->b_dirtyoff); bp->b_dirtyend = max((on + n), bp->b_dirtyend); @@ -1242,8 +1268,22 @@ again: } else { bdwrite(bp); } + + if (error != 0) + break; } while (uio->uio_resid > 0 && n > 0); + if (error != 0) { + if (ioflag & IO_UNIT) { + VATTR_NULL(&vattr); + vattr.va_size = orig_size; + /* IO_SYNC is handled implicitely */ + (void)VOP_SETATTR(vp, &vattr, cred); + uio->uio_offset -= orig_resid - uio->uio_resid; + uio->uio_resid = orig_resid; + } + } + return (error); } Index: /usr/src/sys/fs/nfsclient/nfs_clvfsops.c =================================================================== --- /usr/src/sys/fs/nfsclient/nfs_clvfsops.c (revision 235546) +++ /usr/src/sys/fs/nfsclient/nfs_clvfsops.c (working copy) @@ -1136,7 +1136,8 @@ nfs_mount(struct mount *mp) out: if (!error) { MNT_ILOCK(mp); - mp->mnt_kern_flag |= (MNTK_MPSAFE|MNTK_LOOKUP_SHARED); + mp->mnt_kern_flag |= MNTK_MPSAFE | MNTK_LOOKUP_SHARED | + MNTK_NO_IOPF; MNT_IUNLOCK(mp); } return (error); Index: /usr/src/sys/sys/mount.h =================================================================== --- /usr/src/sys/sys/mount.h (revision 235546) +++ /usr/src/sys/sys/mount.h (working copy) @@ -369,6 +369,9 @@ void __mnt_vnode_markerfree(struct vnode #define MNTK_REFEXPIRE 0x00000020 /* refcount expiring is happening */ #define MNTK_EXTENDED_SHARED 0x00000040 /* Allow shared locking for more ops */ #define MNTK_SHARED_WRITES 0x00000080 /* Allow shared locking for writes */ +#define MNTK_NO_IOPF 0x00000100 /* Disallow page faults during reads + and writes. Filesystem shall properly + handle i/o state on EFAULT. */ #define MNTK_NOASYNC 0x00800000 /* disable async */ #define MNTK_UNMOUNT 0x01000000 /* unmount in progress */ #define MNTK_MWAIT 0x02000000 /* waiting for unmount to finish */ Index: /usr/src/sys/sys/proc.h =================================================================== --- /usr/src/sys/sys/proc.h (revision 235546) +++ /usr/src/sys/sys/proc.h (working copy) @@ -213,6 +213,7 @@ struct thread { struct seltd *td_sel; /* Select queue/channel. */ struct sleepqueue *td_sleepqueue; /* (k) Associated sleep queue. */ struct turnstile *td_turnstile; /* (k) Associated turnstile. */ + struct rl_q_entry *td_rlqe; /* (k) Associated range lock entry. */ struct umtx_q *td_umtxq; /* (c?) Link for when we're blocked. */ lwpid_t td_tid; /* (b) Thread ID. */ sigqueue_t td_sigqueue; /* (c) Sigs arrived, not delivered. */ @@ -311,7 +312,9 @@ struct thread { struct vnet *td_vnet; /* (k) Effective vnet. */ const char *td_vnet_lpush; /* (k) Debugging vnet push / pop. */ struct trapframe *td_intr_frame;/* (k) Frame of the current irq */ - struct proc *td_rfppwait_p; /* (k) The vforked child */ + struct proc *td_rfppwait_p; /* (k) The vforked child */ + struct vm_page **td_ma; /* (k) uio pages held */ + int td_ma_cnt; /* (k) size of *td_ma */ }; struct mtx *thread_lock_block(struct thread *); @@ -419,6 +422,7 @@ do { \ #define TDP_RFPPWAIT 0x02000000 /* Handle RFPPWAIT on syscall exit */ #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */ #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ +#define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */ /* * Reasons that the current thread can not be run yet. Index: /usr/src/sys/sys/vnode.h =================================================================== --- /usr/src/sys/sys/vnode.h (revision 235546) +++ /usr/src/sys/sys/vnode.h (working copy) @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -164,7 +165,8 @@ struct vnode { */ struct vpollinfo *v_pollinfo; /* i Poll events, p for *v_pi */ struct label *v_label; /* MAC label for vnode */ - struct lockf *v_lockf; /* Byte-level lock list */ + struct lockf *v_lockf; /* Byte-level adv lock list */ + struct rangelock v_rl; /* Byte-range lock */ }; #endif /* defined(_KERNEL) || defined(_KVM_VNODE) */ @@ -676,7 +678,17 @@ int vn_extattr_rm(struct vnode *vp, int ioflg, int const char *attrname, struct thread *td); int vn_vget_ino(struct vnode *vp, ino_t ino, int lkflags, struct vnode **rvp); +int vn_io_fault_uiomove(char *data, int xfersize, struct uio *uio); +#define vn_rangelock_unlock(vp, cookie) \ + rangelock_unlock(&(vp)->v_rl, (cookie), VI_MTX(vp)) +#define vn_rangelock_unlock_range(vp, cookie, start, end) \ + rangelock_unlock_range(&(vp)->v_rl, (cookie), (start), (end), \ + VI_MTX(vp)) +#define vn_rangelock_rlock(vp, start, end) \ + rangelock_rlock(&(vp)->v_rl, (start), (end), VI_MTX(vp)) +#define vn_rangelock_wlock(vp, start, end) \ + rangelock_wlock(&(vp)->v_rl, (start), (end), VI_MTX(vp)) int vfs_cache_lookup(struct vop_lookup_args *ap); void vfs_timestamp(struct timespec *); Index: /usr/src/sys/conf/files =================================================================== --- /usr/src/sys/conf/files (revision 235546) +++ /usr/src/sys/conf/files (working copy) @@ -2552,6 +2552,7 @@ kern/kern_priv.c standard kern/kern_proc.c standard kern/kern_prot.c standard kern/kern_racct.c standard +kern/kern_rangelock.c standard kern/kern_rctl.c standard kern/kern_resource.c standard kern/kern_rmlock.c standard Index: /usr/src/sys/ufs/ffs/ffs_vnops.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_vnops.c (revision 235546) +++ /usr/src/sys/ufs/ffs/ffs_vnops.c (working copy) @@ -567,7 +567,7 @@ ffs_read(ap) xfersize = size; } - error = uiomove((char *)bp->b_data + blkoffset, + error = vn_io_fault_uiomove((char *)bp->b_data + blkoffset, (int)xfersize, uio); if (error) break; @@ -738,8 +738,8 @@ ffs_write(ap) if (size < xfersize) xfersize = size; - error = - uiomove((char *)bp->b_data + blkoffset, (int)xfersize, uio); + error = vn_io_fault_uiomove((char *)bp->b_data + blkoffset, + (int)xfersize, uio); /* * If the buffer is not already filled and we encounter an * error while trying to fill it, we have to clear out any Index: /usr/src/sys/ufs/ffs/ffs_vfsops.c =================================================================== --- /usr/src/sys/ufs/ffs/ffs_vfsops.c (revision 235546) +++ /usr/src/sys/ufs/ffs/ffs_vfsops.c (working copy) @@ -1064,7 +1064,7 @@ ffs_mountfs(devvp, mp, td) */ MNT_ILOCK(mp); mp->mnt_kern_flag |= MNTK_MPSAFE | MNTK_LOOKUP_SHARED | - MNTK_EXTENDED_SHARED; + MNTK_EXTENDED_SHARED | MNTK_NO_IOPF; MNT_IUNLOCK(mp); #ifdef UFS_EXTATTR #ifdef UFS_EXTATTR_AUTOSTART Index: /usr/src/sys/kern/vfs_vnops.c =================================================================== --- /usr/src/sys/kern/vfs_vnops.c (revision 235546) +++ /usr/src/sys/kern/vfs_vnops.c (working copy) @@ -65,10 +65,15 @@ __FBSDID("$FreeBSD$"); #include #include +#include +#include +#include #include +#include static fo_rdwr_t vn_read; static fo_rdwr_t vn_write; +static fo_rdwr_t vn_io_fault; static fo_truncate_t vn_truncate; static fo_ioctl_t vn_ioctl; static fo_poll_t vn_poll; @@ -77,8 +82,8 @@ static fo_stat_t vn_statfile; static fo_close_t vn_closefile; struct fileops vnops = { - .fo_read = vn_read, - .fo_write = vn_write, + .fo_read = vn_io_fault, + .fo_write = vn_io_fault, .fo_truncate = vn_truncate, .fo_ioctl = vn_ioctl, .fo_poll = vn_poll, @@ -367,57 +372,56 @@ sequential_heuristic(struct uio *uio, struct file * Package up an I/O request on a vnode into a uio and do it. */ int -vn_rdwr(rw, vp, base, len, offset, segflg, ioflg, active_cred, file_cred, - aresid, td) - enum uio_rw rw; - struct vnode *vp; - void *base; - int len; - off_t offset; - enum uio_seg segflg; - int ioflg; - struct ucred *active_cred; - struct ucred *file_cred; - ssize_t *aresid; - struct thread *td; +vn_rdwr(enum uio_rw rw, struct vnode *vp, void *base, int len, off_t offset, + enum uio_seg segflg, int ioflg, struct ucred *active_cred, + struct ucred *file_cred, ssize_t *aresid, struct thread *td) { struct uio auio; struct iovec aiov; struct mount *mp; struct ucred *cred; + void *rl_cookie; int error, lock_flags; VFS_ASSERT_GIANT(vp->v_mount); + auio.uio_iov = &aiov; + auio.uio_iovcnt = 1; + aiov.iov_base = base; + aiov.iov_len = len; + auio.uio_resid = len; + auio.uio_offset = offset; + auio.uio_segflg = segflg; + auio.uio_rw = rw; + auio.uio_td = td; + error = 0; + if ((ioflg & IO_NODELOCKED) == 0) { + if (rw == UIO_READ) { + rl_cookie = vn_rangelock_rlock(vp, offset, + offset + len); + } else { + rl_cookie = vn_rangelock_wlock(vp, offset, + offset + len); + } mp = NULL; if (rw == UIO_WRITE) { if (vp->v_type != VCHR && (error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) - return (error); + goto out; if (MNT_SHARED_WRITES(mp) || - ((mp == NULL) && MNT_SHARED_WRITES(vp->v_mount))) { + ((mp == NULL) && MNT_SHARED_WRITES(vp->v_mount))) lock_flags = LK_SHARED; - } else { + else lock_flags = LK_EXCLUSIVE; - } - vn_lock(vp, lock_flags | LK_RETRY); } else - vn_lock(vp, LK_SHARED | LK_RETRY); + lock_flags = LK_SHARED; + vn_lock(vp, lock_flags | LK_RETRY); + } else + rl_cookie = NULL; - } ASSERT_VOP_LOCKED(vp, "IO_NODELOCKED with no vp lock held"); - auio.uio_iov = &aiov; - auio.uio_iovcnt = 1; - aiov.iov_base = base; - aiov.iov_len = len; - auio.uio_resid = len; - auio.uio_offset = offset; - auio.uio_segflg = segflg; - auio.uio_rw = rw; - auio.uio_td = td; - error = 0; #ifdef MAC if ((ioflg & IO_NOMACCHECK) == 0) { if (rw == UIO_READ) @@ -429,7 +433,7 @@ int } #endif if (error == 0) { - if (file_cred) + if (file_cred != NULL) cred = file_cred; else cred = active_cred; @@ -444,10 +448,13 @@ int if (auio.uio_resid && error == 0) error = EIO; if ((ioflg & IO_NODELOCKED) == 0) { - if (rw == UIO_WRITE && vp->v_type != VCHR) + VOP_UNLOCK(vp, 0); + if (mp != NULL) vn_finished_write(mp); - VOP_UNLOCK(vp, 0); } + out: + if (rl_cookie != NULL) + vn_rangelock_unlock(vp, rl_cookie); return (error); } @@ -688,29 +695,270 @@ unlock: return (error); } +static const int io_hold_cnt = 16; + /* + * The vn_io_fault() is a wrapper around vn_read() and vn_write() to + * prevent the following deadlock: + * + * Assume that the thread A reads from the vnode vp1 into userspace + * buffer buf1 backed by the pages of vnode vp2. If a page in buf1 is + * currently not resident, then system ends up with the call chain + * vn_read() -> VOP_READ(vp1) -> uiomove() -> [Page Fault] -> + * vm_fault(buf1) -> vnode_pager_getpages(vp2) -> VOP_GETPAGES(vp2) + * which establishes lock order vp1->vn_lock, then vp2->vn_lock. + * If, at the same time, thread B reads from vnode vp2 into buffer buf2 + * backed by the pages of vnode vp1, and some page in buf2 is not + * resident, we get a reversed order vp2->vn_lock, then vp1->vn_lock. + * + * To prevent the lock order reversal and deadlock, vn_io_fault() does + * not allow page faults to happen during VOP_READ() or VOP_WRITE(). + * Instead, it first tries to do the whole range i/o with pagefaults + * disabled. If all pages in the i/o buffer are resident and mapped, + * VOP will succeed (ignoring the genuine filesystem errors). + * Otherwise, we get back EFAULT, and vn_io_fault() falls back to do + * i/o in chunks, with all pages in the chunk prefaulted and held + * using vm_fault_quick_hold_pages(). + * + * Filesystems using this deadlock avoidance scheme should use the + * array of the held pages from uio, saved in the curthread->td_ma, + * instead of doing uiomove(). A helper function + * vn_io_fault_uiomove() converts uiomove request into + * uiomove_fromphys() over td_ma array. + * + * Since vnode locks do not cover the whole i/o anymore , rangelocks + * make the current i/o request atomic with respect to other i/os and + * truncations. + */ +static int +vn_io_fault(struct file *fp, struct uio *uio, struct ucred *active_cred, + int flags, struct thread *td) +{ + vm_page_t ma[io_hold_cnt + 2]; + struct uio *uio_clone, short_uio; + struct iovec short_iovec[1]; + fo_rdwr_t *doio; + struct vnode *vp; + void *rl_cookie; + struct mount *mp; + vm_page_t *prev_td_ma; + int cnt, error, save, saveheld, prev_td_ma_cnt; + vm_offset_t addr, end; + vm_prot_t prot; + size_t len, resid; + ssize_t adv; + + if (uio->uio_rw == UIO_READ) + doio = vn_read; + else + doio = vn_write; + vp = fp->f_vnode; + if (uio->uio_segflg != UIO_USERSPACE || vp->v_type != VREG || + ((mp = vp->v_mount) != NULL && + (mp->mnt_kern_flag & MNTK_NO_IOPF) == 0)) + return (doio(fp, uio, active_cred, flags, td)); + + /* + * The UFS follows IO_UNIT directive and replays back both + * uio_offset and uio_resid if error encountered during the + * operation. But, since the iovec may be already advanced, + * uio is still in the inconsistent state. + * + * Cache a copy of the original uio, which is advanced to redo + * point using UIO_NOCOPY below. + */ + uio_clone = cloneuio(uio); + resid = uio->uio_resid; + + short_uio.uio_segflg = UIO_USERSPACE; + short_uio.uio_rw = uio->uio_rw; + short_uio.uio_td = uio->uio_td; + + if (uio->uio_rw == UIO_READ) { + prot = VM_PROT_WRITE; + rl_cookie = vn_rangelock_rlock(vp, uio->uio_offset, + uio->uio_offset + uio->uio_resid); + } else { + prot = VM_PROT_READ; + if ((fp->f_flag & O_APPEND) != 0 || (flags & FOF_OFFSET) == 0) + /* For appenders, punt and lock the whole range. */ + rl_cookie = vn_rangelock_wlock(vp, 0, OFF_MAX); + else + rl_cookie = vn_rangelock_wlock(vp, uio->uio_offset, + uio->uio_offset + uio->uio_resid); + } + + save = vm_fault_disable_pagefaults(); + error = doio(fp, uio, active_cred, flags, td); + if (error != EFAULT) + goto out; + + uio_clone->uio_segflg = UIO_NOCOPY; + uiomove(NULL, resid - uio->uio_resid, uio_clone); + uio_clone->uio_segflg = uio->uio_segflg; + + saveheld = curthread_pflags_set(TDP_UIOHELD); + prev_td_ma = td->td_ma; + prev_td_ma_cnt = td->td_ma_cnt; + + while (uio_clone->uio_resid != 0) { + len = uio_clone->uio_iov->iov_len; + if (len == 0) { + KASSERT(uio_clone->uio_iovcnt >= 1, + ("iovcnt underflow")); + uio_clone->uio_iov++; + uio_clone->uio_iovcnt--; + continue; + } + + addr = (vm_offset_t)uio_clone->uio_iov->iov_base; + end = round_page(addr + len); + cnt = howmany(end - trunc_page(addr), PAGE_SIZE); + /* + * Perfectly misaligned address and lenght could cause + * both start and end of the chunk to use partial + * page. +2 accounts for such situation. + */ + if (cnt > io_hold_cnt + 2) { + len = io_hold_cnt * PAGE_SIZE; + KASSERT(howmany(round_page(addr + len) - + trunc_page(addr), PAGE_SIZE) <= io_hold_cnt + 2, + ("cnt overflow")); + } + cnt = vm_fault_quick_hold_pages(&td->td_proc->p_vmspace->vm_map, + addr, len, prot, ma, io_hold_cnt + 2); + if (cnt == -1) { + error = EFAULT; + break; + } + short_uio.uio_iov = &short_iovec[0]; + short_iovec[0].iov_base = (void *)addr; + short_uio.uio_iovcnt = 1; + short_uio.uio_resid = short_iovec[0].iov_len = len; + short_uio.uio_offset = uio_clone->uio_offset; + td->td_ma = ma; + td->td_ma_cnt = cnt; + + error = doio(fp, &short_uio, active_cred, flags, td); + vm_page_unhold_pages(ma, cnt); + adv = len - short_uio.uio_resid; + + uio_clone->uio_iov->iov_base = + (char *)uio_clone->uio_iov->iov_base + adv; + uio_clone->uio_iov->iov_len -= adv; + uio_clone->uio_resid -= adv; + uio_clone->uio_offset += adv; + + uio->uio_resid -= adv; + uio->uio_offset += adv; + + if (error != 0 || adv == 0) + break; + } + td->td_ma = prev_td_ma; + td->td_ma_cnt = prev_td_ma_cnt; + curthread_pflags_restore(saveheld); +out: + vm_fault_enable_pagefaults(save); + vn_rangelock_unlock(vp, rl_cookie); + free(uio_clone, M_IOV); + return (error); +} + +/* + * Helper function to perform the requested uiomove operation using + * the held pages for io->uio_iov[0].iov_base buffer instead of + * copyin/copyout. Access to the pages with uiomove_fromphys() + * instead of iov_base prevents page faults that could occur due to + * pmap_collect() invalidating the mapping created by + * vm_fault_quick_hold_pages(), or pageout daemon, page laundry or + * object cleanup revoking the write access from page mappings. + * + * Filesystems specified MNTK_NO_IOPF shall use vn_io_fault_uiomove() + * instead of plain uiomove(). + */ +int +vn_io_fault_uiomove(char *data, int xfersize, struct uio *uio) +{ + struct uio transp_uio; + struct iovec transp_iov[1]; + struct thread *td; + size_t adv; + int error, pgadv; + + td = curthread; + if ((td->td_pflags & TDP_UIOHELD) == 0 || + uio->uio_segflg != UIO_USERSPACE) + return (uiomove(data, xfersize, uio)); + + KASSERT(uio->uio_iovcnt == 1, ("uio_iovcnt %d", uio->uio_iovcnt)); + transp_iov[0].iov_base = data; + transp_uio.uio_iov = &transp_iov[0]; + transp_uio.uio_iovcnt = 1; + if (xfersize > uio->uio_resid) + xfersize = uio->uio_resid; + transp_uio.uio_resid = transp_iov[0].iov_len = xfersize; + transp_uio.uio_offset = 0; + transp_uio.uio_segflg = UIO_SYSSPACE; + /* + * Since transp_iov points to data, and td_ma page array + * corresponds to original uio->uio_iov, we need to invert the + * direction of the i/o operation as passed to + * uiomove_fromphys(). + */ + switch (uio->uio_rw) { + case UIO_WRITE: + transp_uio.uio_rw = UIO_READ; + break; + case UIO_READ: + transp_uio.uio_rw = UIO_WRITE; + break; + } + transp_uio.uio_td = uio->uio_td; + error = uiomove_fromphys(td->td_ma, + ((vm_offset_t)uio->uio_iov->iov_base) & PAGE_MASK, + xfersize, &transp_uio); + adv = xfersize - transp_uio.uio_resid; + pgadv = + (((vm_offset_t)uio->uio_iov->iov_base + adv) >> PAGE_SHIFT) - + (((vm_offset_t)uio->uio_iov->iov_base) >> PAGE_SHIFT); + td->td_ma += pgadv; + KASSERT(td->td_ma_cnt >= pgadv, ("consumed pages %d %d", td->td_ma_cnt, + pgadv)); + td->td_ma_cnt -= pgadv; + uio->uio_iov->iov_base = (char *)uio->uio_iov->iov_base + adv; + uio->uio_iov->iov_len -= adv; + uio->uio_resid -= adv; + uio->uio_offset += adv; + return (error); +} + +/* * File table truncate routine. */ static int -vn_truncate(fp, length, active_cred, td) - struct file *fp; - off_t length; - struct ucred *active_cred; - struct thread *td; +vn_truncate(struct file *fp, off_t length, struct ucred *active_cred, + struct thread *td) { struct vattr vattr; struct mount *mp; struct vnode *vp; + void *rl_cookie; int vfslocked; int error; vp = fp->f_vnode; + + /* + * Lock the whole range for truncation. Otherwise splitted + * i/o might partially happen before, partially after the + * truncation. + */ + rl_cookie = vn_rangelock_wlock(vp, 0, OFF_MAX); vfslocked = VFS_LOCK_GIANT(vp->v_mount); error = vn_start_write(vp, &mp, V_WAIT | PCATCH); - if (error) { - VFS_UNLOCK_GIANT(vfslocked); - return (error); - } + if (error) + goto out1; vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); if (vp->v_type == VDIR) { error = EISDIR; @@ -730,7 +978,9 @@ static int out: VOP_UNLOCK(vp, 0); vn_finished_write(mp); +out1: VFS_UNLOCK_GIANT(vfslocked); + vn_rangelock_unlock(vp, rl_cookie); return (error); } Index: /usr/src/sys/kern/kern_thread.c =================================================================== --- /usr/src/sys/kern/kern_thread.c (revision 235546) +++ /usr/src/sys/kern/kern_thread.c (working copy) @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -205,6 +206,7 @@ thread_init(void *mem, int size, int flags) td->td_sleepqueue = sleepq_alloc(); td->td_turnstile = turnstile_alloc(); + td->td_rlqe = NULL; EVENTHANDLER_INVOKE(thread_init, td); td->td_sched = (struct td_sched *)&td[1]; umtx_thread_init(td); @@ -222,6 +224,7 @@ thread_fini(void *mem, int size) td = (struct thread *)mem; EVENTHANDLER_INVOKE(thread_fini, td); + rlqentry_free(td->td_rlqe); turnstile_free(td->td_turnstile); sleepq_free(td->td_sleepqueue); umtx_thread_fini(td); Index: /usr/src/sys/kern/subr_syscall.c =================================================================== --- /usr/src/sys/kern/subr_syscall.c (revision 235546) +++ /usr/src/sys/kern/subr_syscall.c (working copy) @@ -182,6 +182,12 @@ syscallret(struct thread *td, int error, struct sy KASSERT(td->td_locks == 0, ("System call %s returning with %d locks held", syscallname(p, sa->code), td->td_locks)); + KASSERT((td->td_pflags & TDP_NOFAULTING) == 0, + ("System call %s returning with pagefaults disabled", + syscallname(p, sa->code))); + KASSERT((td->td_pflags & TDP_NOSLEEPING) == 0, + ("System call %s returning with sleep disabled", + syscallname(p, sa->code))); /* * Handle reschedule and other end-of-syscall issues Index: /usr/src/sys/kern/vfs_subr.c =================================================================== --- /usr/src/sys/kern/vfs_subr.c (revision 235546) +++ /usr/src/sys/kern/vfs_subr.c (working copy) @@ -1027,6 +1027,7 @@ alloc: if ((mp->mnt_kern_flag & MNTK_NOKNOTE) != 0) vp->v_vflag |= VV_NOKNOTE; } + rangelock_init(&vp->v_rl); *vpp = vp; return (0); @@ -2468,6 +2469,7 @@ vdropl(struct vnode *vp) /* XXX Elsewhere we detect an already freed vnode via NULL v_op. */ vp->v_op = NULL; #endif + rangelock_destroy(&vp->v_rl); lockdestroy(vp->v_vnlock); mtx_destroy(&vp->v_interlock); mtx_destroy(BO_MTX(bo)); -- Test scenario: datamove4.sh