r235350M, vmcore.353 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r235350M: Sat May 12 21:56:27 CEST 2012 pho@x4.osted.lan:/var/tmp/vmcontention/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) lock order reversal: 1st 0xc1bb505c system map (system map) @ vm/vm_map.c:3723 2nd 0xc130d000 vm object (kernel object) @ vm/vm_fault.c:299 3rd 0xc1bb50e8 system map (system map) @ vm/vm_map.c:1454 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,34313a63,a3435,c1000a39,c0dbde40,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3bd,c12d9048,5ae,c18207c4,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3bd,c1bb50e8,c100de2a,c9582110,c100e4a8,...) at _witness_debugger+0x25 witness_checkorder(c1bb50e8,9,c100e49f,5ae,0,...) at witness_checkorder+0x86f _mtx_lock_flags(c1bb50e8,0,c100e49f,5ae,c596e000,...) at _mtx_lock_flags+0xd9 _vm_map_lock(c1bb508c,c100e49f,5ae,0,a,...) at _vm_map_lock+0x31 vm_map_find(c1bb508c,0,0,0,c1820878,...) at vm_map_find+0x31 kmem_alloc_nofault(c1bb508c,1000,c0fd08e7,eb,100,...) at kmem_alloc_nofault+0x67 vm_radix_node_zone_allocf(c1bb69a0,1000,c18208df,101,c1bb69a0,...) at vm_radix_node_zone_allocf+0x58 keg_alloc_slab(c1bb7608,4,c100d464,871,c182092c,...) at keg_alloc_slab+0xf9 keg_fetch_slab(101,11,c596ff44,c182098c,c0d00311,...) at keg_fetch_slab+0x17f zone_fetch_slab(c1bb69a0,c1bb7600,101,827,1,...) at zone_fetch_slab+0x4c uma_zalloc_arg(c1bb69a0,0,101,c596f4e0,8,...) at uma_zalloc_arg+0x591 vm_radix_insert(c130d030,23480,0,c21fbf80,0,...) at vm_radix_insert+0x1d8 vm_page_insert(c21fbf80,c130d000,23480,0,0,...) at vm_page_insert+0x1b5 vm_page_alloc(c130d000,23480,0,40,c1820bbc,...) at vm_page_alloc+0x51c vm_fault_hold(c1bb5000,e337f000,0,1,0,...) at vm_fault_hold+0x60a vm_fault(c1bb5000,e337f000,0,1,c1bb3a68,...) at vm_fault+0x8b vm_fault_wire(c1bb5000,e3000000,e33c9000,0,0,...) at vm_fault_wire+0x3f vm_map_wire(c1bb5000,e3000000,e33c9000,0,e3000000,...) at vm_map_wire+0x2bc kmem_alloc(c1bb5000,3c9000,c955c770,0,0,...) at kmem_alloc+0x10d vm_ksubmap_init(c131e94c,0,1,1000,0,...) at vm_ksubmap_init+0x64 cpu_startup(0,181ec00,181ec00,181e000,1825000,...) at cpu_startup+0x24f mi_startup() at mi_startup+0xac begin() at begin+0x2c avail memory = 3538612224 (3374 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 621465 free (1321 frags, 77518 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 123324 free (5316 frags, 14751 blocks, 0.5% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 80773709 free (645 frags, 10096633 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 3420947 free (232971 frags, 398497 blocks, 2.3% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 9039002 free (74650 frags, 1120544 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa inet 127.0.0.1 netmask 0xff000000 nd6 options=21 re0: flags=8843 metric 0nk mtu 1500 opti stons=8209bed ether 00:21:8 to5:15:33:7d ine Ut 192.168.1.101 P netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (100baseTX ) status: active fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/gcc46 /usr/local/lib/mysql /usr/local/lib/qt4 /usr/local/lib/virtualbox a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 rpc.umntall: 127.0.0.1: MOUNTPROG: RPC: Program not registered Clearing /tmp (X related). Starting mountd. Starting nfsuserd. Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xcc735058 ufs (ufs) @ kern/vfs_lookup.c:542 2nd 0xe3129140 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 3rd 0xcc7345c8 ufs (ufs) @ kern/vfs_subr.c:2159 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,3a632e73,a303632,0,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3bd,c12d74b8,86f,f40c533c,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3bd,cc7345c8,c0fc2884,c95895a8,c0fe23fa,...) at _witness_debugger+0x25 witness_checkorder(cc7345c8,9,c0fe23f1,86f,0,...) at witness_checkorder+0x86f __lockmgr_args(cc7345c8,80100,cc734634,0,0,...) at __lockmgr_args+0x8c5 ffs_lock(f40c5444,c0ad499c,c0fdf847,5e2,c1108c08,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c11378c0,f40c5444,cae26c30,c1149a20,cc734570,...) at VOP_LOCK1_APV+0xf3 _vn_lock(cc734570,80100,c0fe23f1,86f,eb,...) at _vn_lock+0x78 vget(cc734570,80100,cae26b80,50,0,...) at vget+0xbb vfs_hash_get(cbfa7b70,15ef55,80000,cae26b80,f40c5590,...) at vfs_hash_get+0xed ffs_vgetf(cbfa7b70,15ef55,80000,f40c5590,1,...) at ffs_vgetf+0x49 softdep_sync_buf(cc735000,e31290e0,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(cc735000,1,0,cae26c30,c1108c08,...) at ffs_syncvnode+0x24c ffs_truncate(cc735000,200,0,880,cc07f300,...) at ffs_truncate+0x8b5 ufs_direnter(cc735000,cc734570,f40c5904,f40c5bb8,0,...) at ufs_direnter+0x980 ufs_makeinode(f40c5bb8,c1137e80,f40c5afc,f40c5a58,c0e28b10,...) at ufs_makeinode+0x65d ufs_create(f40c5afc,c102d54e,c1149ca0,cc735000,f40c5b14,...) at ufs_create+0x30 VOP_CREATE_APV(c11378c0,f40c5afc,f40c5bb8,f40c5a94,0,...) at VOP_CREATE_APV+0x100 vn_open_cred(f40c5b78,f40c5c40,1b0,0,cc07f300,...) at vn_open_cred+0x205 vn_open(f40c5b78,f40c5c40,1b0,cc05f0a8,4faed807,...) at vn_open+0x3b kern_openat(cae26b80,ffffff9c,886cf60,0,a03,...) at kern_openat+0x1ec kern_open(cae26b80,886cf60,0,a02,1b0,...) at kern_open+0x35 sys_open(cae26b80,f40c5cec,c1025d9c,c0fdb0af,c1027350,...) at sys_open+0x30 syscall(f40c5d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283d01c3, esp = 0xbfbfc4ec, ebp = 0xbfbfc578 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xcc36bb38 ufs (ufs) @ kern/vfs_mount.c:1254 2nd 0xcc075058 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1834 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,38313a63,a3433,cc38a46c,8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3a4,c12d70e0,72a,f3fd99a8,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3a4,cc075058,c0fcd7a0,c9589408,c1007eb5,...) at _witness_debugger+0x25 witness_checkorder(cc075058,9,c1007eac,72a,cc0750c4,...) at witness_checkorder+0x86f __lockmgr_args(cc075058,80400,cc0750c4,0,0,...) at __lockmgr_args+0x8c5 vop_stdlock(f3fd9aac,c130c880,4,c0fd08e7,eb,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10f8080,f3fd9aac,c130c880,c1149a20,cc075000,...) at VOP_LOCK1_APV+0xf3 _vn_lock(cc075000,80400,c1007eac,72a,0,...) at _vn_lock+0x78 softdep_flushworklist(cc36e2dc,f3fd9b64,cc0b4b80,612,cc0ecba4,...) at softdep_flushworklist+0x47 ffs_sync(cc36e2dc,1,f3fd9bb8,51f,c1bae710,...) at ffs_sync+0x3bf dounmount(cc36e2dc,8080000,cc0b4b80,4a1,f3fd9c3c,...) at dounmount+0x3f8 sys_unmount(cc0b4b80,f3fd9cec,c1025d9c,c0fdb072,c1027350,...) at sys_unmount+0x421 syscall(f3fd9d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280bdd6b, esp = 0xbfbfe63c, ebp = 0xbfbfe708 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes lock order reversal: 1st 0xe3139880 bufwait (bufwait) @ kern/vfs_bio.c:2652 2nd 0xcc024c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,2e687361,38323a63,c1000a34,cc9c0670,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3a4,c12d7470,11c,f40fb7c8,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3a4,cc024c00,c100bdba,c9589610,c100ba4c,...) at _witness_debugger+0x25 witness_checkorder(cc024c00,9,c100ba43,11c,0,...) at witness_checkorder+0x86f _sx_xlock(cc024c00,0,c100ba43,11c,cc7c50e8,...) at _sx_xlock+0x9a ufsdirhash_acquire(e3139820,cc7c50e8,168,e40262b0,f40fb898,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(cc7c50e8,f40fb904,22b0,f40fb884,f40fb888,...) at ufsdirhash_add+0x13 ufs_direnter(cc961000,cc9bd000,f40fb904,f40fbbb8,0,...) at ufs_direnter+0x799 ufs_makeinode(f40fbbb8,c1137e80,f40fbafc,f40fba58,c0e28b10,...) at ufs_makeinode+0x65d ufs_create(f40fbafc,c102d54e,c1149ca0,cc961000,f40fbb14,...) at ufs_create+0x30 VOP_CREATE_APV(c11378c0,f40fbafc,f40fbbb8,f40fba94,0,...) at VOP_CREATE_APV+0x100 vn_open_cred(f40fbb78,f40fbc40,180,0,cbfb2b00,...) at vn_open_cred+0x205 vn_open(f40fbb78,f40fbc40,180,cc05a578,0,...) at vn_open+0x3b kern_openat(cc9c05c0,ffffff9c,bfbfe7ab,0,a02,...) at kern_openat+0x1ec kern_open(cc9c05c0,bfbfe7ab,0,a01,180,...) at kern_open+0x35 sys_open(cc9c05c0,f40fbcec,c1025d9c,c0fdb0af,c1027350,...) at sys_open+0x30 syscall(f40fbd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2829d1c3, esp = 0xbfbfe37c, ebp = 0xbfbfec18 --- May 12 23:39:17 x4 su: pho to root on /dev/pts/0 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 92803 files, 37540272 used, 80773709 free (645 frags, 10096633 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 92803 files, 37540272 used, 80773709 free (645 frags, 10096633 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Sat May 12 23:42:28 CEST 2012 FreeBSD/i386 (x4.osted.lan) (console) login: Expensive timeout(9) function: 0xc08cddb0(0xc132dd60) 0.013211870 s May 12 23:42:39 x4 kernel: pid 1314 (cp), uid 0 inumber 471 on /mnt: out of inodes May 12 23:42:40 x4 kernel: pid 1314 (cp), uid 0 inumber 471 on /mnt: out of inodes May 12 23:42:54 x4 kernel: pid 1372 (cp), uid 0 inumber 475 on /mnt: out of inodes May 12 23:42:55 x4 kernel: pid 1370 (cp), uid 0 inumber 460 on /mnt: out of inodes 20120512 23:43:15 all: nfs4.sh May 12 23:43:15 x4 mountd[1010]: bad exports list line /mnt 20120512 23:44:16 all: truncate4.sh 20120512 23:44:18 all: snap8.sh 20120512 23:44:18 all: lockf2.sh 20120512 23:49:33 all: core.sh 20120512 23:52:01 all: trim4.sh 20120513 00:28:22 all: md.sh May 13 00:28:25 x4 kernel: pid 28998 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:28:33 x4 kernel: pid 28999 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:30:24 x4 kernel: pid 29207 (rw), uid 0 inumber 141 on /mnt: filesystem full May 13 00:30:24 x4 kernel: pid 29198 (rw), uid 0 inumber 11 on /mnt: filesystem full May 13 00:30:26 x4 kernel: pid 29204 (rw), uid 0 inumber 7 on /mnt: filesystem full May 13 00:30:26 x4 kernel: pid 29195 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:30:29 x4 kernel: pid 29196 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:30:29 x4 kernel: pid 29199 (rw), uid 0 inumber 137 on /mnt: filesystem full May 13 00:30:30 x4 kernel: pid 29197 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:30:31 x4 kernel: pid 29208 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:30:32 x4 kernel: pid 29201 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:32:23 x4 kernel: pid 29415 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:32:27 x4 kernel: pid 29416 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:34:25 x4 kernel: pid 29618 (rw), uid 0 inumber 141 on /mnt: filesystem full May 13 00:34:28 x4 kernel: pid 29630 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:34:28 x4 kernel: pid 29631 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:34:31 x4 kernel: pid 29620 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:34:32 x4 kernel: pid 29624 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:36:24 x4 kernel: pid 29843 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:36:26 x4 kernel: pid 29841 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:36:28 x4 kernel: pid 29839 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:36:30 x4 kernel: pid 29842 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:36:33 x4 kernel: pid 29840 (rw), uid 0 inumber 131 on /mnt: filesystem full May 13 00:38:28 x4 kernel: pid 30025 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:38:30 x4 kernel: pid 30023 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 00:38:32 x4 kernel: pid 30026 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:40:27 x4 kernel: pid 30236 (rw), uid 0 inumber 140 on /mnt: filesystem full May 13 00:40:30 x4 kernel: pid 30224 (rw), uid 0 inumber 136 on /mnt: filesystem full May 13 00:40:31 x4 kernel: pid 30223 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 00:42:28 x4 kernel: pid 30439 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 00:42:31 x4 kernel: pid 30441 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:44:25 x4 kernel: pid 30652 (rw), uid 0 inumber 9 on /mnt: filesystem full May 13 00:44:26 x4 kernel: pid 30653 (rw), uid 0 inumber 140 on /mnt: filesystem full May 13 00:44:27 x4 kernel: pid 30655 (rw), uid 0 inumber 140 on /mnt: filesystem full May 13 00:44:27 x4 kernel: pid 30649 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:44:29 x4 kernel: pid 30659 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:44:30 x4 kernel: pid 30645 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:44:30 x4 kernel: pid 30654 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:44:32 x4 kernel: pid 30650 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:46:26 x4 kernel: pid 30869 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:46:31 x4 kernel: pid 30868 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:46:32 x4 kernel: pid 30863 (rw), uid 0 inumber 135 on /mnt: filesystem full May 13 00:46:32 x4 kernel: pid 30867 (rw), uid 0 inumber 134 on /mnt: filesystem full May 13 00:48:23 x4 kernel: pid 31063 (rw), uid 0 inumber 137 on /mnt: filesystem full May 13 00:48:24 x4 kernel: pid 31065 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:48:25 x4 kernel: pid 31070 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:48:27 x4 kernel: pid 31068 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:48:30 x4 kernel: pid 31071 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:48:31 x4 kernel: pid 31069 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:50:23 x4 kernel: pid 31290 (rw), uid 0 inumber 142 on /mnt: filesystem full May 13 00:50:23 x4 kernel: pid 31291 (rw), uid 0 inumber 13 on /mnt: filesystem full May 13 00:50:25 x4 kernel: pid 31289 (rw), uid 0 inumber 4 on /mnt: filesystem full May 13 00:50:26 x4 kernel: pid 31292 (rw), uid 0 inumber 6 on /mnt: filesystem full May 13 00:50:27 x4 kernel: pid 31294 (rw), uid 0 inumber 7 on /mnt: filesystem full May 13 00:50:27 x4 kernel: pid 31287 (rw), uid 0 inumber 6 on /mnt: filesystem full May 13 00:50:30 x4 kernel: pid 31293 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:50:32 x4 kernel: pid 31283 (rw), uid 0 inumber 136 on /mnt: filesystem full May 13 00:50:32 x4 kernel: pid 31275 (rw), uid 0 inumber 130 on /mnt: filesystem full May 13 00:52:26 x4 kernel: pid 31508 (rw), uid 0 inumber 142 on /mnt: filesystem full May 13 00:52:27 x4 kernel: pid 31512 (rw), uid 0 inumber 142 on /mnt: filesystem full May 13 00:52:29 x4 kernel: pid 31501 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:52:31 x4 kernel: pid 31511 (rw), uid 0 inumber 141 on /mnt: filesystem full May 13 00:52:34 x4 kernel: pid 31504 (rw), uid 0 inumber 131 on /mnt: filesystem full May 13 00:54:26 x4 kernel: pid 31714 (rw), uid 0 inumber 12 on /mnt: filesystem full May 13 00:54:26 x4 kernel: pid 31713 (rw), uid 0 inumber 11 on /mnt: filesystem full May 13 00:54:28 x4 kernel: pid 31718 (rw), uid 0 inumber 136 on /mnt: filesystem full May 13 00:54:28 x4 kernel: pid 31710 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:54:30 x4 kernel: pid 31715 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:54:30 x4 kernel: pid 31717 (rw), uid 0 inumber 133 on /mnt: filesystem full May 13 00:54:31 x4 kernel: pid 31723 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:54:31 x4 kernel: pid 31716 (rw), uid 0 inumber 129 on /mnt: filesystem full May 13 00:56:28 x4 kernel: pid 31949 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 00:56:32 x4 kernel: pid 31947 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 00:58:27 x4 kernel: pid 32140 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 00:58:30 x4 kernel: pid 32141 (rw), uid 0 inumber 129 on /mnt: filesystem full lock order reversal: 1st 0xcc072b2c filedesc structure (filedesc structure) @ kern/kern_descrip.c:822 2nd 0xcc54edf0 ufs (ufs) @ kern/vfs_subr.c:4315 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,20296572,656b2040,6b2f6e72,5f6e7265,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3a4,c12d6c68,10db,f40cba68,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3a4,cc54edf0,c0fc2884,c95895a8,c0fe23fa,...) at _witness_debugger+0x25 witness_checkorder(cc54edf0,9,c0fe23f1,10db,cc54ee5c,...) at witness_checkorder+0x86f __lockmgr_args(cc54edf0,80400,cc54ee5c,0,0,...) at __lockmgr_args+0x8c5 ffs_lock(f40cbb70,1,c95823e8,f40cbb54,c0b14c7a,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c11378c0,f40cbb70,c0ad499c,c1149a20,cc54ed98,...) at VOP_LOCK1_APV+0xf3 _vn_lock(cc54ed98,80400,c0fe23f1,10db,f40cbbcc,...) at _vn_lock+0x78 vfs_knllock(cc54ed98,0,c0fcc1e5,728,cc6e7b88,...) at vfs_knllock+0x29 knlist_remove_kq(0,f40cbbec,c0b24169,ccbbb01c,cc6e7b88,...) at knlist_remove_kq+0x85 knlist_remove(ccbbb01c,cc6e7b88,0,f40cbc18,c0a4f555,...) at knlist_remove+0x1b filt_vfsdetach(cc6e7b88,0,c0fcc1e5,832,0,...) at filt_vfsdetach+0x39 knote_fdclose(cae265c0,3,c0fcb92c,336,5,...) at knote_fdclose+0xf5 do_dup(3,cae26808,f40cbd1c,c0e0091e,cae265c0,...) at do_dup+0x351 sys_dup2(cae265c0,f40cbcec,c1025d9c,c0fbbec5,c1027350,...) at sys_dup2+0x28 syscall(f40cbd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (90, FreeBSD ELF32, sys_dup2), eip = 0x280f6caf, esp = 0xbfbfeabc, ebp = 0xbfbfeb08 --- May 13 01:00:23 x4 kernel: pid 32355 (rw), uid 0 inumber 141 on /mnt: filesystem full May 13 01:00:24 x4 kernel: pid 32352 (rw), uid 0 inumber 131 on /mnt: filesystem full May 13 01:00:25 x4 kernel: pid 32369 (rw), uid 0 inumber 140 on /mnt: filesystem full May 13 01:00:26 x4 kernel: pid 32357 (rw), uid 0 inumber 137 on /mnt: filesystem full May 13 01:00:27 x4 kernel: pid 32359 (rw), uid 0 inumber 4 on /mnt: filesystem full May 13 01:00:27 x4 kernel: pid 32366 (rw), uid 0 inumber 6 on /mnt: filesystem full May 13 01:00:28 x4 kernel: pid 32370 (rw), uid 0 inumber 4 on /mnt: filesystem full May 13 01:00:29 x4 kernel: pid 32367 (rw), uid 0 inumber 5 on /mnt: filesystem full May 13 01:00:32 x4 kernel: pid 32364 (rw), uid 0 inumber 4 on /mnt: filesystem full May 13 01:02:25 x4 kernel: pid 32581 (rw), uid 0 inumber 141 on /mnt: filesystem full May 13 01:02:25 x4 kernel: pid 32582 (rw), uid 0 inumber 140 on /mnt: filesystem full May 13 01:02:26 x4 kernel: pid 32594 (rw), uid 0 inumber 11 on /mnt: filesystem full May 13 01:02:26 x4 kernel: pid 32583 (rw), uid 0 inumber 10 on /mnt: filesystem full May 13 01:02:29 x4 kernel: pid 32590 (rw), uid 0 inumber 132 on /mnt: filesystem full May 13 01:02:29 x4 kernel: pid 32580 (rw), uid 0 inumber 4 on /mnt: filesystem full May 13 01:02:30 x4 kernel: pid 32585 (rw), uid 0 inumber 134 on /mnt: filesystem full 20120513 01:04:23 all: rename2.sh 20120513 01:43:22 all: lag.sh 20120513 01:55:56 all: nfsrename.sh May 13 01:55:57 x4 mountd[1010]: bad exports list line /mnt 20120513 01:59:24 all: kevent8.sh 20120513 02:01:48 all: snap.sh lock order reversal: 1st 0xe3351e20 bufwait (bufwait) @ kern/vfs_bio.c:1905 2nd 0xd0f1851c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2297 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,632e746f,3932323a,a37,e3358420,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3a4,c12d7da8,8f9,efbe39c0,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3a4,d0f1851c,c1007974,c9589f68,c10078fe,...) at _witness_debugger+0x25 witness_checkorder(d0f1851c,9,c10078f5,8f9,0,...) at witness_checkorder+0x86f __lockmgr_args(d0f1851c,80900,cc0750c4,0,0,...) at __lockmgr_args+0x8c5 ffs_copyonwrite(cc075000,e336e320,e336e320,4,0,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(cc0750f8,e336e320,3a0,e336e320,4000,...) at ffs_geom_strategy+0xeb bufwrite(e336e320,0,c100ad90,811,1) at bufwrite+0x159 ffs_bufwrite(e336e320,cc778000,efbe3b84,c0cdc7f1,e336e320,...) at ffs_bufwrite+0x290 bawrite(e336e320,e669c000,4000,4000,0,...) at bawrite+0x5c ffs_sbupdate(cc7ba800,3,0,efbe3bc0,c0b24a8a,...) at ffs_sbupdate+0x171 ffs_sync(ccbd9000,3,2,e06,ccbd9000,...) at ffs_sync+0x192 sync_fsync(efbe3c6c,c102ca04,80400,c0fe23f1,6df,...) at sync_fsync+0x156 VOP_FSYNC_APV(c11142c0,efbe3c6c,c0fe23f1,6df,cacf98a0,...) at VOP_FSYNC_APV+0x100 sync_vnode(c1304bb4,c1304ba0,3e8,783,4e20,...) at sync_vnode+0x16b sched_sync(0,efbe3d28,c0fcc8d0,3d8,cbedb598,...) at sched_sync+0x282 fork_exit(c0b2ad70,0,efbe3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbe3d60, ebp = 0 --- lock order reversal: 1st 0xd0f1851c snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:966 2nd 0xd430d9dc ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1627 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,a373236,a00,1,f42eb680,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3a4,c12d6848,65b,f42eb718,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3a4,d430d9dc,c0fc2884,c95895a8,c10078fe,...) at _witness_debugger+0x25 witness_checkorder(d430d9dc,9,c10078f5,65b,0,...) at witness_checkorder+0x86f __lockmgr_args(d430d9dc,80000,0,0,0,...) at __lockmgr_args+0x8c5 ffs_snapremove(d430d984,c1007833,f42eb848,c11a9db4,c119c340,...) at ffs_snapremove+0x11f ffs_truncate(d430d984,0,0,c00,0,...) at ffs_truncate+0x647 ufs_inactive(f42ebad4,c102cf1d,eb,c11378c0,f42ebaa8,...) at ufs_inactive+0x1f6 VOP_INACTIVE_APV(c11378c0,f42ebad4,c0fe23f1,9bf,f42ebb08,...) at VOP_INACTIVE_APV+0x100 vinactive(d430d984,cf8962e0,c0fe23f1,8be,0,...) at vinactive+0x127 vputx(f42ebc34,c0b2eab3,d430d984,ffffffdf,d430d984,...) at vputx+0x370 vput(d430d984,ffffffdf,d430d984,f42ebb8c,6dda780,...) at vput+0x10 kern_unlinkat(cf8962e0,ffffff9c,bfbfea72,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(cf8962e0,bfbfea72,0,f42ebd1c,c0e0091e,...) at kern_unlink+0x2f sys_unlink(cf8962e0,f42ebcec,c1025d9c,c0fdb0c3,c1027350,...) at sys_unlink+0x22 syscall(f42ebd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x28160e1f, esp = 0xbfbfe79c, ebp = 0xbfbfe818 --- 20120513 02:21:38 all: sem.sh 20120513 02:21:59 all: jail.sh 20120513 02:22:03 all: nullfs2.sh 20120513 02:37:18 all: jail2.sh 20120513 02:37:20 all: fpu.sh 20120513 02:39:58 all: mac.sh 20120513 02:39:58 all: suj5.sh 20120513 02:41:30 all: umountf6.sh May 13 02:41:37 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:41:37 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument : : May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:44:12 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument 20120513 02:44:14 all: nullfs8.sh lock order reversal: 1st 0xd4750310 newnfs (newnfs) @ kern/vfs_syscalls.c:4185 2nd 0xe331f9c0 bufwait (bufwait) @ kern/vfs_bio.c:1905 3rd 0xd433c1b4 newnfs (newnfs) @ kern/vfs_subr.c:2159 KDB: stack backtrace: db_trace_self_wrapper(c0fd68d7,a3935,a3500,cdd9f000,5,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0ac240b,c0fda3bd,c12d26b0,86f,f44db414,...) at kdb_backtrace+0x2a _witness_debugger(c0fda3bd,d433c1b4,c0fbcecb,c9589e30,c0fe23fa,...) at _witness_debugger+0x25 witness_checkorder(d433c1b4,9,c0fe23f1,86f,0,...) at witness_checkorder+0x86f __lockmgr_args(d433c1b4,80100,d433c220,0,0,...) at __lockmgr_args+0x8c5 nfs_lock1(f44db518,c0ad499c,246,c12d26b4,c95827f8,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c10f91a0,f44db518,cd5c5390,c1149a20,d433c15c,...) at VOP_LOCK1_APV+0xf3 _vn_lock(d433c15c,80100,c0fe23f1,86f,eb,...) at _vn_lock+0x78 vget(d433c15c,80100,cd5c52e0,50,0,...) at vget+0xbb vfs_hash_get(d16fd5b8,277b10c1,80000,cd5c52e0,f44db5e0,...) at vfs_hash_get+0xed nfscl_nget(d16fd5b8,d47502b8,cff976a0,f44db794,cd5c52e0,...) at nfscl_nget+0x8d nfsrpc_readdirplus(d47502b8,f44db980,f44db93c,cbfb2680,cd5c52e0,...) at nfsrpc_readdirplus+0x13ce ncl_readdirplusrpc(d47502b8,f44db980,cbfb2680,cd5c52e0,d5a4d40c,...) at ncl_readdirplusrpc+0xec ncl_doio(d47502b8,e331f960,cbfb2680,cd5c52e0,0,...) at ncl_doio+0x28d ncl_bioread(d47502b8,f44dbbe4,0,cbfb2680,c0e2219a,...) at ncl_bioread+0xc4d nfs_readdir(f44dbc04,c102c9c1,200400,c0fe3ac7,1059,...) at nfs_readdir+0x192 VOP_READDIR_APV(c10f91a0,f44dbc04,c0fe3ac7,1059,f44dbc2c,...) at VOP_READDIR_APV+0x100 kern_getdirentries(cd5c52e0,5,8893000,1000,f44dbc58,...) at kern_getdirentries+0x230 sys_getdirentries(cd5c52e0,f44dbcec,c1025d9c,c0fdb063,c1027350,...) at sys_getdirentries+0x31 syscall(f44dbd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (196, FreeBSD ELF32, sys_getdirentries), eip = 0x280dcedf, esp = 0xbfbfe70c, ebp = 0xbfbfe738 --- 20120513 02:46:35 all: nullfs7.sh 20120513 02:46:36 all: crossmp.sh May 13 02:46:40 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:46:43 x4 mountd[1010]: can't delete exports for /mnt9: Device busy May 13 02:46:43 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument : : May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:47:08 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument fsync: giving up on dirty 0xd54e4c3c: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd1744e00 flags (VI(0x200)) v_object 0xd188e1bc ref 0 pages 10 lock type devfs: EXCL by thread 0xd5da65c0 (pid 32562, umount, tid 2101029) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md19a May 13 02:47:17 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:47:17 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:47:17 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:47:17 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:47:17 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:47:18 x4 mountd[1010]: can't delete exports for /mnt14: Device busy May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:47:19 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument fsync: giving up on dirty 0xd4810828: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd16dc900 flags (VI(0x200)) v_object 0xd555ade0 ref 0 pages 10 lock type devfs: EXCL by thread 0xcf8945c0 (pid 34586, umount, tid 100275) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md6a May 13 02:47:27 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:47:27 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:47:29 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:47:29 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:47:29 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument : : May 13 02:49:03 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:49:08 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:49:10 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:10 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument fsync: giving up on dirty 0xd5aaa414: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xce1be600 flags (VI(0x200)) v_object 0xd5991818 ref 0 pages 10 lock type devfs: EXCL by thread 0xccc4c2e0 (pid 49641, umount, tid 100186) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md10a May 13 02:49:16 x4 mountd[1010]: can't delete exports for /mnt7: Device busy May 13 02:49:17 x4 mountd[1010]: can't delete exports for /mnt7: Device busy May 13 02:49:19 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:49:21 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:23 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:26 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:28 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument fsync: giving up on dirty 0xd4eed570: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd170a900 flags (VI(0x200)) v_object 0xd5652378 ref 0 pages 10 lock type devfs: EXCL by thread 0xcd5c5000 (pid 51758, umount, tid 100402) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md16a May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:32 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:34 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:49:35 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:49:35 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:35 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:49:35 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:49:37 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument fsync: giving up on dirty 0xd54d8d98: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd1745900 flags (VI(0x200)) v_object 0xd2bffe74 ref 0 pages 10 lock type devfs: EXCL by thread 0xd5de68a0 (pid 53770, umount, tid 2101060) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md17a May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:49:52 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:49:54 x4 mountd[1010]: can't delete exports for /mnt11: Device busy May 13 02:49:57 x4 mountd[1010]: can't delete exports for /mnt9: Device busy May 13 02:49:58 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:49:58 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:49:58 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:49:58 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:49:58 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:09 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:12 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:50:12 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:50:13 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument fsync: giving up on dirty 0xd54d8d98: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd1745900 flags (VI(0x200)) v_object 0xd2bffe74 ref 0 pages 10 lock type devfs: EXCL by thread 0xcd5c48a0 (pid 57863, umount, tid 100404) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md17a May 13 02:50:16 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:18 x4 mountd[1010]: can't delete exports for /mnt6: Device busy May 13 02:50:18 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:19 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:19 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:19 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:19 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:19 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:20 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:50:20 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:20 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:20 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:20 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:21 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:23 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:24 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:24 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:24 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:24 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:24 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument fsync: giving up on dirty 0xd54702b8: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd1752c00 flags (VI(0x200)) v_object 0xd26c5128 ref 0 pages 10 lock type devfs: EXCL by thread 0xcf88d2e0 (pid 59455, umount, tid 100298) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md18a May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt15: Invalid argument May 13 02:50:27 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:50:29 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:29 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:29 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:29 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:29 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:30 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:31 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:31 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:31 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:31 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt13: Device busy May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt9: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt19: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt10: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt17: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:38 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:40 x4 mountd[1010]: can't delete exports for /mnt7: Invalid argument May 13 02:50:40 x4 mountd[1010]: can't delete exports for /mnt8: Invalid argument May 13 02:50:40 x4 mountd[1010]: can't delete exports for /mnt16: Invalid argument May 13 02:50:40 x4 mountd[1010]: can't delete exports for /mnt11: Invalid argument May 13 02:50:40 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument fsync: giving up on dirty 0xd4810828: tag devfs, type VCHR usecount 1, writecount 0, refcount 5 mountedhere 0xd16dc900 flags (VI(0x200)) v_object 0xd555ade0 ref 0 pages 10 lock type devfs: EXCL by thread 0xcf8962e0 (pid 61693, umount, tid 100264) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0b14b95 at vop_stdlock+0x65 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0cdceb7 at ffs_sync+0x467 #5 0xc0b1fb48 at dounmount+0x3f8 #6 0xc0b20231 at sys_unmount+0x421 #7 0xc0e0091e at syscall+0x2fe #8 0xc0deabe1 at Xint0x80_syscall+0x21 dev md6a May 13 02:50:43 x4 mountd[1010]: can't delete exports for /mnt8: Device busy May 13 02:50:43 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 02:50:43 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 02:50:43 x4 mountd[1010]: can't delete exports for /mnt14: Invalid argument May 13 02:50:43 x4 mountd[1010]: can't delete exports for /mnt18: Invalid argument : : May 13 03:23:54 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument May 13 03:23:54 x4 mountd[1010]: can't delete exports for /mnt13: Invalid argument May 13 03:23:56 x4 mountd[1010]: can't delete exports for /mnt6: Invalid argument May 13 03:24:14 x4 mountd[1010]: can't delete exports for /mnt5: Invalid argument May 13 03:24:22 x4 mountd[1010]: can't delete exports for /mnt12: Invalid argument 20120513 03:24:40 all: context2.sh 20120513 03:27:19 all: pread.sh 20120513 03:28:06 all: pageout.sh 20120513 03:42:10 all: suj19.sh May 13 03:42:11 x4 kernel: pid 73425 (rw), uid 1004 inumber 98308 on /mnt: filesystem full May 13 03:44:13 x4 kernel: pid 73769 (rw), uid 1004 inumber 65675 on /mnt: filesystem full May 13 03:46:14 x4 kernel: pid 74051 (rw), uid 1004 inumber 65736 on /mnt: filesystem full May 13 03:48:13 x4 kernel: pid 74318 (rw), uid 1004 inumber 98332 on /mnt: filesystem full May 13 03:50:13 x4 kernel: pid 74591 (rw), uid 1004 inumber 84 on /mnt: filesystem full May 13 03:50:16 x4 kernel: pid 74596 (rw), uid 1004 inumber 14 on /mnt: filesystem full May 13 03:52:14 x4 kernel: pid 74860 (rw), uid 1004 inumber 65560 on /mnt: filesystem full May 13 03:54:14 x4 kernel: pid 75120 (rw), uid 1004 inumber 65643 on /mnt: filesystem full May 13 03:54:16 x4 kernel: pid 75123 (rw), uid 1004 inumber 8 on /mnt: filesystem full May 13 03:56:21 x4 kernel: pid 75435 (rw), uid 1004 inumber 12 on /mnt: filesystem full May 13 03:58:27 x4 kernel: pid 75727 (rw), uid 1004 inumber 65546 on /mnt: filesystem full 20120513 04:02:28 all: devfs2.sh 20120513 04:02:29 all: suj14.sh May 13 04:03:59 x4 kernel: pid 76864 (cp), uid 0 inumber 51826 on /mnt: filesystem full May 13 04:04:00 x4 kernel: pid 76867 (mkdir), uid 0 inumber 51844 on /mnt: filesystem full May 13 04:04:01 x4 kernel: pid 76884 (cp), uid 0 inumber 51843 on /mnt: filesystem full 20120513 04:04:05 all: devfd.sh 20120513 04:04:17 all: quota8.sh 20120513 04:07:38 all: symlink2.sh 20120513 04:07:45 all: truncate2.sh 20120513 04:10:47 all: gjournal.sh GEOM_JOURNAL: Journal 2930982199: md5 contains data. GEOM_JOURNAL: Journal 2930982199: md5 contains journal. GEOM_JOURNAL: Journal md5 clean. GEOM_JOURNAL: BIO_FLUSH not supported by md5. panic: kmem_malloc(131072): kmem_map too small: 324399104 total allocated cpuid = 1 KDB: enter: panic [ thread pid 78934 tid 100169 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 78934 tid 100169 td 0xcc0d1000 kdb_enter(c0fd2704,c0fd2704,c100e2a4,f41645c4,1,...) at kdb_enter+0x3a panic(c100e2a4,20000,1355f000,c100e29e,7d0,...) at panic+0x18c kmem_malloc(c1bb5118,20000,2,f4164628,c0cfe290,...) at kmem_malloc+0x280 page_alloc(0,20000,f416461b,2,2000113,...) at page_alloc+0x27 uma_large_malloc(20000,2,0,2,2,...) at uma_large_malloc+0x50 malloc(20000,cc1c4130,2,113,20000,...) at malloc+0x14d gj_malloc(ddb19000,d14fe000,20000,3af18000,0,...) at gj_malloc+0x99 g_journal_new_bio(3af38000,0,7bff6600,0,de19b000,...) at g_journal_new_bio+0x62 g_journal_insert(3af38000,0,7bff6600,0,de19b000,...) at g_journal_insert+0xe0 g_journal_flush(cc679c10,0,cc1c2704,87f,bb8,...) at g_journal_flush+0x37c g_journal_worker(cc679c00,f4164d28,c0fcc8d0,3d8,ccd7bb30,...) at g_journal_worker+0x16a6 fork_exit(cc1bff50,cc679c00,f4164d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf4164d60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x6d4a00 curthread = 0xc9622000: pid 12 "swi4: clock" curpcb = 0xc93fed80 fpcurthread = none idlethread = 0xc95df8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x828aa00 curthread = 0xcc0d1000: pid 78934 "g_journal md5" curpcb = 0xf4164d80 fpcurthread = none idlethread = 0xc95df5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x828da00 curthread = 0xc95df2e0: pid 11 "idle: cpu2" curpcb = 0xc93f1d80 fpcurthread = none idlethread = 0xc95df2e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x8290a00 curthread = 0xc95df000: pid 11 "idle: cpu3" curpcb = 0xc93f4d80 fpcurthread = none idlethread = 0xc95df000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 78944 (dd) thread 0xcf8235c0 (100281) exclusive lockmgr ufs (ufs) r = 0 (0xd437b058) locked @ kern/vfs_vnops.c:645 Process 78940 (md6) thread 0xcf60b8a0 (2100982) exclusive lockmgr bufwait (bufwait) r = 0 (0xe3106da0) locked @ kern/vfs_bio.c:1905 exclusive lockmgr ufs (ufs) r = 0 (0xd77779dc) locked @ dev/md/md.c:621 db:0:alllocks> show lockedvnods Locked vnodes 0xd7777984: tag ufs, type VREG usecount 1, writecount 1, refcount 1698 mountedhere 0 flags (VV_MD|VI(0x200)) v_object 0xd2264784 ref 0 pages 34704 lock type ufs: EXCL by thread 0xcf60b8a0 (pid 78940, md6, tid 2100982) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0ce2b51 at ffs_lock+0xa1 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0784998 at mdstart_vnode+0x498 #5 0xc07835c0 at md_kthread+0x1c0 #6 0xc0a56ed8 at fork_exit+0xb8 #7 0xc0deabf4 at fork_trampoline+0x8 ino 4, on dev md5.journal 0xd437b000: tag ufs, type VREG usecount 1, writecount 1, refcount 1742 mountedhere 0 flags (VI(0x200)) v_object 0xdde43534 ref 0 pages 34872 lock type ufs: EXCL by thread 0xcf8235c0 (pid 78944, dd, tid 100281) #0 0xc0a6c68d at __lockmgr_args+0xd6d #1 0xc0ce2b51 at ffs_lock+0xa1 #2 0xc0e276d3 at VOP_LOCK1_APV+0xf3 #3 0xc0b37e98 at _vn_lock+0x78 #4 0xc0b38ce6 at vn_write+0x176 #5 0xc0ad6bf5 at dofilewrite+0x95 #6 0xc0ad7268 at kern_writev+0x68 #7 0xc0ad737f at sys_write+0x4f #8 0xc0e0091e at syscall+0x2fe #9 0xc0deabe1 at Xint0x80_syscall+0x21 ino 4, on dev md6a db:0:lockedvnods> show mount 0xcbfa6b70 /dev/ad4s1a on / (ufs) 0xcbfa7000 devfs on /dev (devfs) 0xcc36e5b8 /dev/ad4s1f on /home (ufs) 0xcc36e000 /dev/ad4s1d on /usr (ufs) 0xcbfa7b70 /dev/ad4s1e on /var (ufs) 0xcbfa7894 procfs on /proc (procfs) 0xccbd9000 /dev/label/tmp on /tmp (ufs) 0xcbfa62dc /dev/md5.journal on /mnt (ufs) 0xd8dd2b70 /dev/md6a on /media (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 78960 1228 1226 1001 S nanslp 0xc118e9e4 sleep 78945 78912 1413 0 S+ piperd 0xd8de5c80 egrep 78944 78912 1413 0 D+ wdrain 0xc1304864 dd 78940 0 0 0 DL biowr 0xe3106d40 [md6] 78934 0 0 0 RL CPU 1 [g_journal md5] 78932 0 0 0 DL suspwt 0xcbfa6328 [g_journal switcher] 78930 0 0 0 DL mdwait 0xd8e02800 [md5] 78912 1413 1413 0 S+ wait 0xcca312cc sh 1413 1208 1413 0 S+ wait 0xcc0cc000 sh 1299 1 1299 0 Ss+ ttyin 0xcae33870 getty 1298 1 1298 0 Ss+ ttyin 0xcae33a70 getty 1297 1 1297 0 Ss+ ttyin 0xc970d270 getty 1296 1 1296 0 Ss+ ttyin 0xc970d470 getty 1295 1 1295 0 Ss+ ttyin 0xc970d670 getty 1294 1 1294 0 Ss+ ttyin 0xc970d870 getty 1293 1 1293 0 Ss+ ttyin 0xc970da70 getty 1292 1 1292 0 Ss+ ttyin 0xc970dc70 getty 1291 1 1291 0 Ss+ ttyin 0xc970de70 getty 1274 1 1274 0 Ss select 0xcc06e1e4 inetd 1249 1 1249 0 Ss nanslp 0xc118e9e4 watchdogd 1229 1226 1226 1001 S piperd 0xcc0cb960 awk 1228 1226 1226 1001 S wait 0xcc96c2cc sh 1227 1225 1227 1001 Ss+ select 0xcc081764 top 1226 1224 1226 1001 SWs wait 0xcc96c598 sh 1225 1217 1217 1001 S select 0xcc06e6a4 sshd 1224 1218 1218 1001 S select 0xcc082264 sshd 1223 1222 1223 1001 Ss kqread 0xcbf9e580 tail 1222 1216 1216 1001 S select 0xcc7824e4 sshd 1218 1148 1218 0 Ss select 0xcc081024 sshd 1217 1148 1217 0 Ss select 0xcc782764 sshd 1216 1148 1216 0 Ss select 0xcac867e4 sshd 1208 1205 1208 0 SW+ wait 0xcc0cc598 bash 1205 1204 1205 0 SW+ pause 0xcc96b324 csh 1204 1199 1204 1001 SW+ wait 0xcc0cf000 su 1199 1198 1199 1001 SWs+ wait 0xcc0af864 bash 1198 1196 1196 1001 S select 0xcbfa53a4 sshd 1196 1148 1196 0 Ss select 0xcc06e2a4 sshd 1159 1 1159 0 Ss nanslp 0xc118e9e4 cron 1154 1 1154 25 Ss pause 0xcc5628bc sendmail 1151 1 1151 0 Ss select 0xcc06e964 sendmail 1148 1 1148 0 Ss select 0xcc7822a4 sshd 1116 1 1116 0 Ss select 0xcac7ad64 moused 1092 1 1092 0 Ss select 0xcac86da4 ntpd 1037 1032 1032 0 S (threaded) nfsd 100125 S rpcsvc 0xcaca9e50 nfsd: service 100124 S rpcsvc 0xcaca9e90 nfsd: service 100123 S rpcsvc 0xcc02a010 nfsd: service 100104 S rpcsvc 0xcaca8850 nfsd: master 1032 1 1032 0 Ss select 0xcc081b24 nfsd 1030 1026 1026 0 S select 0xcc0810a4 nfsuserd 1029 1026 1026 0 S select 0xcc06e9e4 nfsuserd 1028 1026 1026 0 S select 0xcc0810e4 nfsuserd 1027 1026 1026 0 S select 0xcbfa50a4 nfsuserd 1026 1 1026 0 SWs pause 0xcc0aeb88 nfsuserd 1010 1 1010 0 Ss select 0xcc06ede4 mountd 976 1 976 0 Ss select 0xcc0821e4 rpcbind 961 1 961 0 Ss select 0xcbfa5de4 syslogd 838 1 838 0 Ss select 0xcc06ec64 devd 19 0 0 0 DL sdflush 0xc130c8c0 [softdepflush] 18 0 0 0 DL vlruwt 0xcbedb2cc [vnlru] 17 0 0 0 DL syncer 0xc1304bb4 [syncer] 16 0 0 0 DL psleep 0xc1304848 [bufdaemon] 9 0 0 0 DL pgzero 0xc130e76c [pagezero] 8 0 0 0 DL psleep 0xc130e348 [vmdaemon] 7 0 0 0 DL psleep 0xc130e310 [pagedaemon] 6 0 0 0 DL ccb_scan 0xc1151354 [xpt_thrd] 5 0 0 0 DL waiting_ 0xc13080d8 [sctp_iterator] 4 0 0 0 DL - 0xcac7223c [fdc0] 3 0 0 0 SL - 0xcad2c000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100059 D - 0xcacfed34 [usbus5] 100058 D - 0xcacfed04 [usbus5] 100057 D - 0xcacfecd4 [usbus5] 100056 D - 0xcacfeca4 [usbus5] 100054 D - 0xcacefb5c [usbus4] 100053 D - 0xcacefb2c [usbus4] 100052 D - 0xcacefafc [usbus4] 100051 D - 0xcacefacc [usbus4] 100050 D - 0xcaceab5c [usbus3] 100049 D - 0xcaceab2c [usbus3] 100048 D - 0xcaceaafc [usbus3] 100047 D - 0xcaceaacc [usbus3] 100046 D - 0xcace5b5c [usbus2] 100045 D - 0xcace5b2c [usbus2] 100044 D - 0xcace5afc [usbus2] 100043 D - 0xcace5acc [usbus2] 100041 D - 0xcacddb5c [usbus1] 100040 D - 0xcacddb2c [usbus1] 100039 D - 0xcacddafc [usbus1] 100038 D - 0xcacddacc [usbus1] 100036 D - 0xcacd5b5c [usbus0] 100035 D - 0xcacd5b2c [usbus0] 100034 D - 0xcacd5afc [usbus0] 100033 D - 0xcacd5acc [usbus0] 2 0 0 0 DL ctl_work 0xc97c4000 [ctl_thrd] 14 0 0 0 DL - 0xc118d7e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100016 D - 0xc118a8a8 [g_down] 100015 D - 0xc118a8a4 [g_up] 100014 D - 0xc118a89c [g_event] 12 0 0 0 WL (threaded) [intr] 100067 I [irq12: psm0] 100066 I [irq1: atkbd0] 100064 I [swi0: uart] 100061 I [irq20: fwohci0] 100060 I [irq14: ata0] 100055 I [irq19: ehci0] 100042 I [irq18: ohci2 ohci4] 100037 I [irq17: ohci1 ohci3] 100032 I [irq16: hdac1 ohci0] 100031 I [irq22: ahci0] 100030 I [irq257: re0] 100029 I [irq256: hdac0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: +] 100018 I [swi2: cambio] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xc95ddb30 [init] 10 0 0 0 DL audit_wo 0xc130c000 [audit] 0 0 0 0 DLs (threaded) [kernel] 100069 D - 0xc118d7e4 [deadlkres] 100068 D - 0xc971ddc0 [mca taskq] 100062 D - 0xc971de00 [fw0_taskq] 100027 D - 0xc971de40 [kqueue taskq] 100026 D - 0xc971de80 [ffs_trim taskq] 100025 D - 0xc971e000 [acpi_task_2] 100024 D - 0xc971e000 [acpi_task_1] 100023 D - 0xc971e000 [acpi_task_0] 100020 D - 0xc971e0c0 [thread taskq] 100013 D - 0xc96e9000 [firmware taskq] 100000 D sched 0xc118aa60 [swapper] db:0:ps> allt Tracing command sleep pid 78960 tid 2100995 td 0xd5de95c0 sched_switch(d5de95c0,0,104,1a3,1baa3bc3,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(d5de95c0,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(2711,c0ac8c30,d5de95c0,2,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c118e9e4,6c,c0fd3c8b,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c118e9e4,0,16c,c0fd3c8b,2711,...) at _sleep+0x34f kern_nanosleep(d5de95c0,f4669c48,f4669c50,a,0,...) at kern_nanosleep+0xc1 sys_nanosleep(d5de95c0,f4669cec,c1025d9c,c0fdba0e,c1027350,...) at sys_nanosleep+0x6f syscall(f4669d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2814fdc3, esp = 0xbfbfec4c, ebp = 0xbfbfec98 --- Tracing command egrep pid 78945 tid 100369 td 0xcf8268a0 sched_switch(cf8268a0,0,104,1a3,13cffb4e,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(cf8268a0,0,c0fd82c0,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(d8de5c80,5c,c0fdac47,100,0,...) at sleepq_wait_sig+0x17 _sleep(d8de5c80,d8de5df8,15c,c0fdac47,0,...) at _sleep+0x385 pipe_read(cd001000,f4453c3c,cc07f580,0,cf8268a0,...) at pipe_read+0x427 dofileread(f4453c3c,ffffffff,ffffffff,0,f4453c18,...) at dofileread+0x9e kern_readv(cf8268a0,0,f4453c3c,f4453c5c,1,...) at kern_readv+0x68 sys_read(cf8268a0,f4453cec,c1025d9c,c0fdb13a,c1027350,...) at sys_read+0x4f syscall(f4453d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b87c3, esp = 0xbfbfe62c, ebp = 0xbfbfe6b8 --- Tracing command dd pid 78944 tid 100281 td 0xcf8235c0 sched_switch(cf8235c0,0,104,1a3,febee4ce,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(cf8235c0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1304864,54,c0fdf85f,0,0,...) at sleepq_wait+0x63 _sleep(c1304864,c1304868,54,c0fdf85f,0,...) at _sleep+0x39c waitrunningbufspace(d437b0f8,e30e7f40,3a0,e3325ae0,c2a39c00,...) at waitrunningbufspace+0x6a bufwrite(e30e7f40,4,f431e990,c0b12da7,e30e7f40,...) at bufwrite+0x190 bawrite(e30e7f40,e30e8048,20,3af,0,...) at bawrite+0x5c cluster_wbuild(d437b000,8000,1104,0,4,...) at cluster_wbuild+0x837 cluster_write(d437b000,e3325ae0,8820000,0,7f,...) at cluster_write+0x6d6 ffs_write(f431eba4,c102d483,d437b000,d437b000,f431eb5c,...) at ffs_write+0x53a VOP_WRITE_APV(c11378c0,f431eba4,d437b000,285,0,...) at VOP_WRITE_APV+0x174 vn_write(cc05fd20,f431ec3c,cc07f580,0,cf8235c0,...) at vn_write+0x2bc dofilewrite(f431ec3c,ffffffff,ffffffff,0,f431ec18,...) at dofilewrite+0x95 kern_writev(cf8235c0,4,f431ec3c,f431ec5c,1,...) at kern_writev+0x68 sys_write(cf8235c0,f431ecec,c1025d9c,c0e3242f,c1027350,...) at sys_write+0x4f syscall(f431ed28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281757a3, esp = 0xbfbfe7dc, ebp = 0xbfbfe808 --- Tracing command md6 pid 78940 tid 2100982 td 0xcf60b8a0 sched_switch(cf60b8a0,0,104,1a3,fef69801,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(cf60b8a0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(e3106d40,5c,c0fdfbf7,0,0,...) at sleepq_wait+0x63 _sleep(e3106d40,c959c844,5c,c0fdfbf7,0,...) at _sleep+0x39c bwait(e3106d40,5c,c0fdfbf7,e3106d40,f4642af8,...) at bwait+0x6f bufwait(e3106d40,e3106d40,2000,2000,0,...) at bufwait+0x48 bufwrite(e3106d40,2000,f4642c6c,2000,cc07f580,...) at bufwrite+0x165 ffs_write(f4642c8c,c102d483,d7777984,d7777984,f4642c14,...) at ffs_write+0x4a5 VOP_WRITE_APV(c11378c0,f4642c8c,c0f72275,26d,c0a90b01,...) at VOP_WRITE_APV+0x174 mdstart_vnode(cc35a000,cc3248e8,c0f72275,2fe,0,...) at mdstart_vnode+0x4da md_kthread(cc35a000,f4642d28,c0fcc8d0,3d8,d03642cc,...) at md_kthread+0x1c0 fork_exit(c0783400,cc35a000,f4642d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf4642d60, ebp = 0 --- Tracing command g_journal md5 pid 78934 tid 100169 td 0xcc0d1000 kdb_enter(c0fd2704,c0fd2704,c100e2a4,f41645c4,1,...) at kdb_enter+0x3a panic(c100e2a4,20000,1355f000,c100e29e,7d0,...) at panic+0x18c kmem_malloc(c1bb5118,20000,2,f4164628,c0cfe290,...) at kmem_malloc+0x280 page_alloc(0,20000,f416461b,2,2000113,...) at page_alloc+0x27 uma_large_malloc(20000,2,0,2,2,...) at uma_large_malloc+0x50 malloc(20000,cc1c4130,2,113,20000,...) at malloc+0x14d gj_malloc(ddb19000,d14fe000,20000,3af18000,0,...) at gj_malloc+0x99 g_journal_new_bio(3af38000,0,7bff6600,0,de19b000,...) at g_journal_new_bio+0x62 g_journal_insert(3af38000,0,7bff6600,0,de19b000,...) at g_journal_insert+0xe0 g_journal_flush(cc679c10,0,cc1c2704,87f,bb8,...) at g_journal_flush+0x37c g_journal_worker(cc679c00,f4164d28,c0fcc8d0,3d8,ccd7bb30,...) at g_journal_worker+0x16a6 fork_exit(cc1bff50,cc679c00,f4164d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf4164d60, ebp = 0 --- Tracing command g_journal switcher pid 78932 tid 2100963 td 0xcf60e2e0 sched_switch(cf60e2e0,0,104,1a3,e683248b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,77,...) at mi_switch+0x1ff sleepq_switch(cf60e2e0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(cbfa6328,77,c0fe3d5a,0,0,...) at sleepq_wait+0x63 _sleep(cbfa6328,cbfa62dc,277,c0fe3d5a,0,...) at _sleep+0x39c vfs_write_suspend(cbfa62dc,2,1,b5d,2710,...) at vfs_write_suspend+0xef g_journal_switcher(cc1c40c0,f4609d28,c0fcc8d0,3d8,cdf45598,...) at g_journal_switcher+0xb9e fork_exit(cc1bb5d0,cc1c40c0,f4609d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf4609d60, ebp = 0 --- Tracing command md5 pid 78930 tid 2100948 td 0xd5d6b5c0 sched_switch(d5d6b5c0,0,104,1a3,fb6a714f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(d5d6b5c0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(d8e02800,5c,c0f72322,0,0,...) at sleepq_wait+0x63 _sleep(d8e02800,d8e02820,25c,c0f72322,0,...) at _sleep+0x39c md_kthread(d8e02800,f45dcd28,c0fcc8d0,3d8,ce1452cc,...) at md_kthread+0x11a fork_exit(c0783400,d8e02800,f45dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf45dcd60, ebp = 0 --- Tracing command sh pid 78912 tid 100350 td 0xcf6138a0 sched_switch(cf6138a0,0,104,1a3,1352a629,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cf6138a0,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cca312cc,6c,c0fdb1f8,100,0,...) at sleepq_wait_sig+0x17 _sleep(cca312cc,cca31354,16c,c0fdb1f8,0,...) at _sleep+0x385 kern_wait(cf6138a0,ffffffff,f4412c58,2,0,...) at kern_wait+0x597 sys_wait4(cf6138a0,f4412cec,c1025d9c,c0fdb1d7,c1027350,...) at sys_wait4+0x3b syscall(f4412d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2814f6eb, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command sh pid 1413 tid 100110 td 0xcc0d15c0 sched_switch(cc0d15c0,0,104,1a3,a102464b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cc0d15c0,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc0cc000,6c,c0fdb1f8,100,0,...) at sleepq_wait_sig+0x17 _sleep(cc0cc000,cc0cc088,16c,c0fdb1f8,0,...) at _sleep+0x385 kern_wait(cc0d15c0,ffffffff,f406bc58,2,0,...) at kern_wait+0x597 sys_wait4(cc0d15c0,f406bcec,c1025d9c,c0fdb1d7,cacc1b80,...) at sys_wait4+0x3b syscall(f406bd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2814f6eb, esp = 0xbfbfe6ec, ebp = 0xbfbfe708 --- Tracing command getty pid 1299 tid 100167 td 0xccc4cb80 sched_switch(ccc4cb80,0,104,1a3,5207c83f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ccc4cb80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,ccc4cb80,ccc4cb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cae33870,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cae33870,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(cae33800,cae33870,f415ec3c,1,0,...) at tty_wait+0x71 ttydisc_read(cae33800,f415ec3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cae27100,f415ec3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc03d498,f415ec3c,c95d9b80,0,ccc4cb80,...) at devfs_read_f+0x7e dofileread(f415ec3c,ffffffff,ffffffff,0,f415ec18,...) at dofileread+0x9e kern_readv(ccc4cb80,0,f415ec3c,f415ec5c,1,...) at kern_readv+0x68 sys_read(ccc4cb80,f415ecec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f415ed28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1298 tid 100166 td 0xccc4d000 sched_switch(ccc4d000,0,104,1a3,51f2bc08,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ccc4d000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,ccc4d000,ccc4d000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cae33a70,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cae33a70,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(cae33a00,cae33a70,f415bc3c,1,0,...) at tty_wait+0x71 ttydisc_read(cae33a00,f415bc3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cae27000,f415bc3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc0701c0,f415bc3c,c95d9b80,0,ccc4d000,...) at devfs_read_f+0x7e dofileread(f415bc3c,ffffffff,ffffffff,0,f415bc18,...) at dofileread+0x9e kern_readv(ccc4d000,0,f415bc3c,f415bc5c,1,...) at kern_readv+0x68 sys_read(ccc4d000,f415bcec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f415bd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1297 tid 100165 td 0xccc4d2e0 sched_switch(ccc4d2e0,0,104,1a3,51f155d6,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(ccc4d2e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,ccc4d2e0,ccc4d2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970d270,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970d270,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970d200,c970d270,f4158c3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970d200,f4158c3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0ce00,f4158c3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc03dbd0,f4158c3c,c95d9b80,0,ccc4d2e0,...) at devfs_read_f+0x7e dofileread(f4158c3c,ffffffff,ffffffff,0,f4158c18,...) at dofileread+0x9e kern_readv(ccc4d2e0,0,f4158c3c,f4158c5c,1,...) at kern_readv+0x68 sys_read(ccc4d2e0,f4158cec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f4158d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1296 tid 100164 td 0xcc0d58a0 sched_switch(cc0d58a0,0,104,1a3,51f4af2b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d58a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,cc0d58a0,cc0d58a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970d470,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970d470,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970d400,c970d470,f4155c3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970d400,f4155c3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0cd00,f4155c3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc03d930,f4155c3c,c95d9b80,0,cc0d58a0,...) at devfs_read_f+0x7e dofileread(f4155c3c,ffffffff,ffffffff,0,f4155c18,...) at dofileread+0x9e kern_readv(cc0d58a0,0,f4155c3c,f4155c5c,1,...) at kern_readv+0x68 sys_read(cc0d58a0,f4155cec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f4155d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1295 tid 100163 td 0xcc0d5b80 sched_switch(cc0d5b80,0,104,1a3,5205e3ed,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d5b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,cc0d5b80,cc0d5b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970d670,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970d670,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970d600,c970d670,f4152c3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970d600,f4152c3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0cc00,f4152c3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc03d038,f4152c3c,c95d9b80,0,cc0d5b80,...) at devfs_read_f+0x7e dofileread(f4152c3c,ffffffff,ffffffff,0,f4152c18,...) at dofileread+0x9e kern_readv(cc0d5b80,0,f4152c3c,f4152c5c,1,...) at kern_readv+0x68 sys_read(cc0d5b80,f4152cec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f4152d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1294 tid 100162 td 0xcc786000 sched_switch(cc786000,0,104,1a3,5202170f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc786000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,cc786000,cc786000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970d870,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970d870,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970d800,c970d870,f414fc3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970d800,f414fc3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0c900,f414fc3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc05a498,f414fc3c,c95d9b80,0,cc786000,...) at devfs_read_f+0x7e dofileread(f414fc3c,ffffffff,ffffffff,0,f414fc18,...) at dofileread+0x9e kern_readv(cc786000,0,f414fc3c,f414fc5c,1,...) at kern_readv+0x68 sys_read(cc786000,f414fcec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f414fd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1293 tid 100161 td 0xcc7862e0 sched_switch(cc7862e0,0,104,1a3,520b6307,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc7862e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,cc7862e0,cc7862e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970da70,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970da70,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970da00,c970da70,f414cc3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970da00,f414cc3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0c800,f414cc3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc05a5b0,f414cc3c,c95d9b80,0,cc7862e0,...) at devfs_read_f+0x7e dofileread(f414cc3c,ffffffff,ffffffff,0,f414cc18,...) at dofileread+0x9e kern_readv(cc7862e0,0,f414cc3c,f414cc5c,1,...) at kern_readv+0x68 sys_read(cc7862e0,f414ccec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f414cd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1292 tid 100160 td 0xcc7865c0 sched_switch(cc7865c0,0,104,1a3,520a02fa,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc7865c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,cc7865c0,cc7865c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970dc70,0,c0fdd080,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970dc70,c118c710,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970dc00,c970dc70,f4149c3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970dc00,f4149c3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(cac0c700,f4149c3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc03cb28,f4149c3c,c95d9b80,0,cc7865c0,...) at devfs_read_f+0x7e dofileread(f4149c3c,ffffffff,ffffffff,0,f4149c18,...) at dofileread+0x9e kern_readv(cc7865c0,0,f4149c3c,f4149c5c,1,...) at kern_readv+0x68 sys_read(cc7865c0,f4149cec,c1025d9c,c0e371f2,c1027350,...) at sys_read+0x4f syscall(f4149d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command getty pid 1291 tid 100159 td 0xcc7868a0 sched_switch(cc7868a0,0,104,1a3,97b2d417,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc7868a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,c970de04,0,c0fd08e7,cc7868a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c970de70,0,f4146ae8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c970de70,c970de04,c0fdca58,527,0,...) at _cv_wait_sig+0x260 tty_wait(c970de00,c970de70,f4146c3c,1,0,...) at tty_wait+0x71 ttydisc_read(c970de00,f4146c3c,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c95c6100,f4146c3c,0,0,1,...) at ttydev_read+0xaa devfs_read_f(cc05f690,f4146c3c,c95d9b80,0,cc7868a0,...) at devfs_read_f+0x7e dofileread(f4146c3c,ffffffff,ffffffff,0,f4146c18,...) at dofileread+0x9e kern_readv(cc7868a0,0,f4146c3c,f4146c5c,1,...) at kern_readv+0x68 sys_read(cc7868a0,f4146cec,c1025d9c,c0e371f2,0,...) at sys_read+0x4f syscall(f4146d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281877c3, esp = 0xbfbfed6c, ebp = 0xbfbfeda8 --- Tracing command inetd pid 1274 tid 100155 td 0xcc7885c0 sched_switch(cc7885c0,0,104,1a3,478b9018,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc7885c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc06e1d0,0,c0fd08e7,cc7885c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc06e1e4,0,f413aa50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc06e1e4,cc06e1d0,c0fdaa18,6ae,cc05fe38,...) at _cv_wait_sig+0x260 seltdwait(cc05fe38,58,c95d9b80,cc7885c0,0,...) at seltdwait+0xa2 kern_select(cc7885c0,8,bfbfdf90,0,0,0,20,ccbc0864,4) at kern_select+0x655 sys_select(cc7885c0,f413acec,c1025d9c,c0fdb244,0,...) at sys_select+0x66 syscall(f413ad28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28199743, esp = 0xbfbfdf4c, ebp = 0xbfbfedd8 --- Tracing command watchdogd pid 1249 tid 100117 td 0xcae26b80 sched_switch(cae26b80,0,104,1a3,9db18348,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cae26b80,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0ac8c30,cae26b80,3,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c118e9e4,6c,c0fd3c8b,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c118e9e4,0,16c,c0fd3c8b,ea61,...) at _sleep+0x34f kern_nanosleep(cae26b80,f40c5c48,f40c5c50,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(cae26b80,f40c5cec,c1025d9c,c0e371f2,c119b640,...) at sys_nanosleep+0x6f syscall(f40c5d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2817edc3, esp = 0xbfbfecac, ebp = 0xbfbfecd8 --- Tracing command awk pid 1229 tid 100101 td 0xcc02f2e0 sched_switch(cc02f2e0,0,104,1a3,1b887709,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(cc02f2e0,0,c0fd82c0,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc0cb960,5c,c0fdac47,100,0,...) at sleepq_wait_sig+0x17 _sleep(cc0cb960,cc0cbad8,15c,c0fdac47,0,...) at _sleep+0x385 pipe_read(cc03c498,f4028c3c,cbfb2800,0,cc02f2e0,...) at pipe_read+0x427 dofileread(f4028c3c,ffffffff,ffffffff,0,f4028c18,...) at dofileread+0x9e kern_readv(cc02f2e0,0,f4028c3c,f4028c5c,1,...) at kern_readv+0x68 sys_read(cc02f2e0,f4028cec,c1025d9c,c0e3242f,c1027350,...) at sys_read+0x4f syscall(f4028d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281aa7c3, esp = 0xbfbfe92c, ebp = 0xbfbfe948 --- Tracing command sh pid 1228 tid 100130 td 0xcc0b02e0 sched_switch(cc0b02e0,0,104,1a3,1b954d5b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cc0b02e0,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc96c2cc,6c,c0fdb1f8,100,0,...) at sleepq_wait_sig+0x17 _sleep(cc96c2cc,cc96c354,16c,c0fdb1f8,0,...) at _sleep+0x385 kern_wait(cc0b02e0,ffffffff,f40efc58,2,0,...) at kern_wait+0x597 sys_wait4(cc0b02e0,f40efcec,c1025d9c,c0fdb1d7,c119c9c0,...) at sys_wait4+0x3b syscall(f40efd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2814f6eb, esp = 0xbfbfe92c, ebp = 0xbfbfe948 --- Tracing command top pid 1227 tid 100137 td 0xcc9bfb80 sched_switch(cc9bfb80,0,104,1a3,9a6ba388,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc9bfb80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f4104a14,c0a747fa,cc081750,0,cc9bfb80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc081764,0,f4104a50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc081764,cc081750,3e9,6ae,cc03c0e0,...) at _cv_timedwait_sig+0x270 seltdwait(f4104c08,f4104c10,cc07fa00,cc9bfb80,335b1b0a,...) at seltdwait+0x8a kern_select(cc9bfb80,2,bfbfeb9c,0,0,f4104c54,20,1,0) at kern_select+0x655 sys_select(cc9bfb80,f4104cec,c1025d9c,c0fa5962,c1027350,...) at sys_select+0x66 syscall(f4104d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281df743, esp = 0xbfbfeb3c, ebp = 0xbfbfec98 --- Tracing command sh pid 1226 tid 100129 td 0xcc0b05c0 Tracing command sshd pid 1225 tid 100136 td 0xcc9c0000 sched_switch(cc9c0000,0,104,1a3,9a6d1658,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc9c0000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc06e690,0,c0fd08e7,cc9c0000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc06e6a4,0,f4101a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc06e6a4,cc06e690,c0fdaa18,6ae,cc05ac78,...) at _cv_wait_sig+0x260 seltdwait(cc05ac78,58,cc06d180,cc9c0000,cc9c00b0,...) at seltdwait+0xa2 kern_select(cc9c0000,a,8810238,8810220,0,0,20,cc96b000,0) at kern_select+0x655 sys_select(cc9c0000,f4101cec,c1025d9c,c0fdb345,c1027350,...) at sys_select+0x66 syscall(f4101d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d8743, esp = 0xbfbfde1c, ebp = 0xbfbfde68 --- Tracing command sshd pid 1224 tid 100132 td 0xcc9c0b80 sched_switch(cc9c0b80,0,104,1a3,a0f8247b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc9c0b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc082250,0,c0fd08e7,cc9c0b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc082264,0,f40f5a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc082264,cc082250,c0fdaa18,6ae,cc05fc08,...) at _cv_wait_sig+0x260 seltdwait(cc05fc08,58,cc07f900,cc9c0b80,cc9c0c30,...) at seltdwait+0xa2 kern_select(cc9c0b80,c,8810238,8810220,0,0,20,cc96bb30,0) at kern_select+0x655 sys_select(cc9c0b80,f40f5cec,c1025d9c,c0fdb345,c1027350,...) at sys_select+0x66 syscall(f40f5d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d8743, esp = 0xbfbfde1c, ebp = 0xbfbfde68 --- Tracing command tail pid 1223 tid 100119 td 0xcae265c0 sched_switch(cae265c0,0,104,1a3,7c94c7e1,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,68,...) at mi_switch+0x1ff sleepq_switch(cae265c0,0,c0fd82c0,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(3e9,c0ac8c30,cae265c0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cbf9e580,68,c0fcc482,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(cbf9e580,cbf9e580,168,c0fcc482,3e9,...) at _sleep+0x34f kern_kevent(cae265c0,4,0,1,f40cbc3c,...) at kern_kevent+0x374 sys_kevent(cae265c0,f40cbcec,c1025d9c,c0e3242f,c119b640,...) at sys_kevent+0x19e syscall(f40cbd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (363, FreeBSD ELF32, sys_kevent), eip = 0x28153cdb, esp = 0xbfbfeb0c, ebp = 0xbfbfebc8 --- Tracing command sshd pid 1222 tid 100103 td 0xcc0d4b80 sched_switch(cc0d4b80,0,104,1a3,df4a1def,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d4b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc7824d0,0,c0fd08e7,cc0d4b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc7824e4,0,f4037a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc7824e4,cc7824d0,c0fdaa18,6ae,cc05f930,...) at _cv_wait_sig+0x260 seltdwait(cc05f930,58,cc07f880,cc0d4b80,cc0d4c30,...) at seltdwait+0xa2 kern_select(cc0d4b80,c,8810238,8810220,0,0,20,cc0cf598,0) at kern_select+0x655 sys_select(cc0d4b80,f4037cec,c1025d9c,c0fdb345,c1027350,...) at sys_select+0x66 syscall(f4037d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d8743, esp = 0xbfbfde1c, ebp = 0xbfbfde68 --- Tracing command sshd pid 1218 tid 100118 td 0xcae268a0 sched_switch(cae268a0,0,104,1a3,48228129,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae268a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc081010,0,c0fd08e7,cae268a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc081024,0,f40c8ab4,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc081024,cc081010,c0fdaa18,6ae,f40c8b3c,...) at _cv_wait_sig+0x260 seltdwait(cc07202c,c0fdaa18,5ab,cae268a0,0,...) at seltdwait+0xa2 sys_poll(cae268a0,f40c8cec,c1025d9c,c0e3242f,cbee82e0,...) at sys_poll+0x321 syscall(f40c8d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x284700eb, esp = 0xbfbfde7c, ebp = 0xbfbfdeb8 --- Tracing command sshd pid 1217 tid 100091 td 0xcc0d55c0 sched_switch(cc0d55c0,0,104,1a3,4869ebdf,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d55c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc782750,0,c0fd08e7,cc0d55c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc782764,0,f3ffeab4,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc782764,cc782750,c0fdaa18,6ae,f3ffeb3c,...) at _cv_wait_sig+0x260 seltdwait(cc0c422c,c0fdaa18,5ab,cc0d55c0,0,...) at seltdwait+0xa2 sys_poll(cc0d55c0,f3ffecec,c1025d9c,c0e3242f,c1027350,...) at sys_poll+0x321 syscall(f3ffed28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x284700eb, esp = 0xbfbfde7c, ebp = 0xbfbfdeb8 --- Tracing command sshd pid 1216 tid 100092 td 0xcc0d52e0 sched_switch(cc0d52e0,0,104,1a3,4713dcb5,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d52e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cac867d0,0,c0fd08e7,cc0d52e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cac867e4,0,f4002ab4,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cac867e4,cac867d0,c0fdaa18,6ae,f4002b3c,...) at _cv_wait_sig+0x260 seltdwait(cc07222c,c0fdaa18,5ab,cc0d52e0,0,...) at seltdwait+0xa2 sys_poll(cc0d52e0,f4002cec,c1025d9c,c0e3242f,cbee82e0,...) at sys_poll+0x321 syscall(f4002d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x284700eb, esp = 0xbfbfde7c, ebp = 0xbfbfdeb8 --- Tracing command bash pid 1208 tid 100108 td 0xcc0d1b80 Tracing command csh pid 1205 tid 100135 td 0xcc9c02e0 Tracing command su pid 1204 tid 100105 td 0xcc0d45c0 Tracing command bash pid 1199 tid 100083 td 0xcc0b4b80 Tracing command sshd pid 1198 tid 100102 td 0xcc0d5000 sched_switch(cc0d5000,0,104,1a3,9f6f42cc,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d5000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cbfa5390,0,c0fd08e7,cc0d5000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cbfa53a4,0,f4032a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cbfa53a4,cbfa5390,c0fdaa18,6ae,cc03ce38,...) at _cv_wait_sig+0x260 seltdwait(cc03ce38,58,cbfb2c00,cc0d5000,cc0d50b0,...) at seltdwait+0xa2 kern_select(cc0d5000,c,8810238,8810220,0,0,20,cc0cf864,0) at kern_select+0x655 sys_select(cc0d5000,f4032cec,c1025d9c,c0fdb345,c1027350,...) at sys_select+0x66 syscall(f4032d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d8743, esp = 0xbfbfde1c, ebp = 0xbfbfde68 --- Tracing command sshd pid 1196 tid 100109 td 0xcc0d18a0 sched_switch(cc0d18a0,0,104,1a3,6b6b2547,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d18a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc06e290,0,c0fd08e7,cc0d18a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc06e2a4,0,f4067ab4,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc06e2a4,cc06e290,c0fdaa18,6ae,f4067b3c,...) at _cv_wait_sig+0x260 seltdwait(cc0e5a2c,c0fdaa18,5ab,cc0d18a0,696,...) at seltdwait+0xa2 sys_poll(cc0d18a0,f4067cec,c1025d9c,c0fbccd1,c1027350,...) at sys_poll+0x321 syscall(f4067d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x284700eb, esp = 0xbfbfde7c, ebp = 0xbfbfdeb8 --- Tracing command cron pid 1159 tid 100127 td 0xcc0b0b80 sched_switch(cc0b0b80,0,104,1a3,69721bb0,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cc0b0b80,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(d2f1,c0ac8c30,cc0b0b80,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c118e9e4,6c,c0fd3c8b,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c118e9e4,0,16c,c0fd3c8b,d2f1,...) at _sleep+0x34f kern_nanosleep(cc0b0b80,f40e6c48,f40e6c50,36,0,...) at kern_nanosleep+0xc1 sys_nanosleep(cc0b0b80,f40e6cec,c1025d9c,c0fdb64e,14,...) at sys_nanosleep+0x6f syscall(f40e6d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x28171dc3, esp = 0xbfbfec3c, ebp = 0xbfbfec68 --- Tracing command sendmail pid 1154 tid 100115 td 0xcbee62e0 sched_switch(cbee62e0,0,104,1a3,4f9ae1a8,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(cbee62e0,0,c0fd82c0,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc5628bc,74,c0e5900e,100,0,...) at sleepq_wait_sig+0x17 _sleep(cc5628bc,cc5628ec,174,c0e5900e,0,...) at _sleep+0x385 kern_sigsuspend(cbee62e0,0,0,0,0,...) at kern_sigsuspend+0xba sys_sigsuspend(cbee62e0,f40bfcec,c1025d9c,c0fdb0f7,c119b640,...) at sys_sigsuspend+0x4d syscall(f40bfd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x28321c2b, esp = 0xbfbfcf8c, ebp = 0xbfbfcfb8 --- Tracing command sendmail pid 1151 tid 100114 td 0xcbee65c0 sched_switch(cbee65c0,0,104,1a3,19fc71cd,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cbee65c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f40bca14,c0a747fa,cc06e950,0,cbee65c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc06e964,0,f40bca50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc06e964,cc06e950,1389,6ae,ccff8a48,...) at _cv_timedwait_sig+0x270 seltdwait(f40bcc08,f40bcc10,c95daa00,cbee65c0,40,...) at seltdwait+0x8a kern_select(cbee65c0,5,bfbfc500,0,0,f40bcc54,20,5,0) at kern_select+0x655 sys_select(cbee65c0,f40bccec,c1025d9c,c0fdb5d4,14,...) at sys_select+0x66 syscall(f40bcd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283d8743, esp = 0xbfbfc46c, ebp = 0xbfbfcf98 --- Tracing command sshd pid 1148 tid 100081 td 0xcbee82e0 sched_switch(cbee82e0,0,104,1a3,47b3e933,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cbee82e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc782290,0,c0fd08e7,cbee82e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc7822a4,0,f3fcda50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc7822a4,cc782290,c0fdaa18,6ae,cc03d6c8,...) at _cv_wait_sig+0x260 seltdwait(cc03d6c8,58,cbfb2b80,cbee82e0,f3fcdb14,...) at seltdwait+0xa2 kern_select(cbee82e0,9,8810168,0,0,0,20,cac0b864,a) at kern_select+0x655 sys_select(cbee82e0,f3fcdcec,c1025d9c,c0fbccd1,c1027350,...) at sys_select+0x66 syscall(f3fcdd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284d8743, esp = 0xbfbfdedc, ebp = 0xbfbfede8 --- Tracing command moused pid 1116 tid 100097 td 0xcc0b32e0 sched_switch(cc0b32e0,0,104,1a3,562818ea,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0b32e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cac7ad50,0,c0fd08e7,cc0b32e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cac7ad64,0,f4017a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cac7ad64,cac7ad50,c0fdaa18,6ae,cc05fa10,...) at _cv_wait_sig+0x260 seltdwait(cc05fa10,58,c95d9b80,cc0b32e0,f6,...) at seltdwait+0xa2 kern_select(cc0b32e0,400,bfbfe9f4,0,0,0,20,cc0ae000,1) at kern_select+0x655 sys_select(cc0b32e0,f4017cec,c1025d9c,c0e371f2,c1027350,...) at sys_select+0x66 syscall(f4017d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a6743, esp = 0xbfbfe98c, ebp = 0xbfbfeb18 --- Tracing command ntpd pid 1092 tid 100113 td 0xcbee68a0 sched_switch(cbee68a0,0,104,1a3,77f73545,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cbee68a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cac86d90,0,c0fd08e7,cbee68a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cac86da4,0,f40b9a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cac86da4,cac86d90,c0fdaa18,6ae,cc05f428,...) at _cv_wait_sig+0x260 seltdwait(cc05f428,58,c95d9b80,cbee68a0,0,...) at seltdwait+0xa2 kern_select(cbee68a0,1c,bfbfecd8,0,0,0,20,cc563000,0) at kern_select+0x655 sys_select(cbee68a0,f40b9cec,c1025d9c,c0fdba20,e,...) at sys_select+0x66 syscall(f40b9d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28356743, esp = 0xbfbfecac, ebp = 0xbfbfed78 --- Tracing command nfsd pid 1037 tid 100125 td 0xcc788b80 sched_switch(cc788b80,0,104,1a3,bb8e4601,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc788b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f40ddbdc,c0a747fa,cbc36e80,0,cc788b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(caca9e50,0,f40ddc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(caca9e50,cbc36e80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(f40ddd14,c0a56ed8,cbc36e80,f40ddd28,c0fcc8d0,...) at svc_run_internal+0x356 svc_thread_start(cbc36e80,f40ddd28,c0fcc8d0,3d8,cc0cf2cc,...) at svc_thread_start+0x10 fork_exit(c0c84f70,cbc36e80,f40ddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1037 tid 100124 td 0xcc789000 sched_switch(cc789000,0,104,1a3,bb055202,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc789000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f40dabdc,c0a747fa,cbc36e80,0,cc789000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(caca9e90,0,f40dac18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(caca9e90,cbc36e80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(f40dad14,c0a56ed8,cbc36e80,f40dad28,c0fcc8d0,...) at svc_run_internal+0x356 svc_thread_start(cbc36e80,f40dad28,c0fcc8d0,3d8,cc0cf2cc,...) at svc_thread_start+0x10 fork_exit(c0c84f70,cbc36e80,f40dad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1037 tid 100123 td 0xcc7892e0 sched_switch(cc7892e0,0,104,1a3,bb0598bd,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc7892e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f40d7bdc,c0a747fa,cbc36e80,0,cc7892e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc02a010,0,f40d7c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc02a010,cbc36e80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(f40d7d14,c0a56ed8,cbc36e80,f40d7d28,c0fcc8d0,...) at svc_run_internal+0x356 svc_thread_start(cbc36e80,f40d7d28,c0fcc8d0,3d8,cc0cf2cc,...) at svc_thread_start+0x10 fork_exit(c0c84f70,cbc36e80,f40d7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x2e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1037 tid 100104 td 0xcc0d48a0 sched_switch(cc0d48a0,0,104,1a3,7812b504,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0d48a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f404448c,c0a747fa,cbc36e80,0,cc0d48a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(caca8850,0,f40444c8,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(caca8850,cbc36e80,1388,3b1,f4044510,...) at _cv_timedwait_sig+0x270 svc_run_internal(cc0d4a50,14,c1002afa,c0fbe1db,f4044808,...) at svc_run_internal+0x356 svc_run(cbc36e80,0,c0fbe1b6,19f,c12d7920,...) at svc_run+0x7b nfsrvd_nfsd(cc0d48a0,f4044808,c,f7b,c100e49f,...) at nfsrvd_nfsd+0x1c3 nfssvc_nfsd(cc0d48a0,f4044cec,0,cc0d48a0,f4044d1c,...) at nfssvc_nfsd+0x177 sys_nfssvc(cc0d48a0,f4044cec,c1025d9c,c0fdb5d4,c1027350,...) at sys_nfssvc+0xeb syscall(f4044d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280be86b, esp = 0xbfbfe81c, ebp = 0xbfbfea88 --- Tracing command nfsd pid 1032 tid 100088 td 0xcc030000 sched_switch(cc030000,0,104,1a3,f0de2bbf,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc030000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc081b10,0,c0fd08e7,cc030000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc081b24,0,f3fefa50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc081b24,cc081b10,c0fdaa18,6ae,cc05f7a8,...) at _cv_wait_sig+0x260 seltdwait(cc05f7a8,58,c95d9b80,cc030000,58,...) at seltdwait+0xa2 kern_select(cc030000,7,bfbfeca4,0,0,0,20,cc02b000,7) at kern_select+0x655 sys_select(cc030000,f3fefcec,c1025d9c,c0fbccd1,c1027350,...) at sys_select+0x66 syscall(f3fefd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28174743, esp = 0xbfbfea8c, ebp = 0xbfbfedd8 --- Tracing command nfsuserd pid 1030 tid 100084 td 0xcc0b48a0 sched_switch(cc0b48a0,0,104,1a3,7f9cac43,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0b48a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f3fdea14,c0a747fa,cc081090,0,cc0b48a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc0810a4,0,f3fdea50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc0810a4,cc081090,7531,6ae,cc05f1f8,...) at _cv_timedwait_sig+0x270 seltdwait(f3fdec08,f3fdec10,c95d9b80,cc0b48a0,c0a74363,...) at seltdwait+0x8a kern_select(cc0b48a0,4,bfbfebb4,0,0,f3fdec54,20,1e,0) at kern_select+0x655 sys_select(cc0b48a0,f3fdecec,c1025d9c,c0fdb32e,ced505c0,...) at sys_select+0x66 syscall(f3fded28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28173743, esp = 0xbfbfeb0c, ebp = 0xbfbfec48 --- Tracing command nfsuserd pid 1029 tid 100082 td 0xcc0b5000 sched_switch(cc0b5000,0,104,1a3,c011811f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0b5000,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f3fd5a14,c0a747fa,cc06e9d0,0,cc0b5000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc06e9e4,0,f3fd5a50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc06e9e4,cc06e9d0,7531,6ae,cc05f1f8,...) at _cv_timedwait_sig+0x270 seltdwait(f3fd5c08,f3fd5c10,c95d9b80,cc0b5000,0,...) at seltdwait+0x8a kern_select(cc0b5000,4,bfbfebb4,0,0,f3fd5c54,20,1e,0) at kern_select+0x655 sys_select(cc0b5000,f3fd5cec,c1025d9c,c0fdb32e,c1027350,...) at sys_select+0x66 syscall(f3fd5d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28173743, esp = 0xbfbfeb0c, ebp = 0xbfbfec48 --- Tracing command nfsuserd pid 1028 tid 100086 td 0xcc0305c0 sched_switch(cc0305c0,0,104,1a3,7f653f29,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0305c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f3fe7a14,c0a747fa,cc0810d0,0,cc0305c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc0810e4,0,f3fe7a50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc0810e4,cc0810d0,7531,6ae,cc05f1f8,...) at _cv_timedwait_sig+0x270 seltdwait(f3fe7c08,f3fe7c10,c95d9b80,cc0305c0,f3fe7ac4,...) at seltdwait+0x8a kern_select(cc0305c0,4,bfbfebb4,0,0,f3fe7c54,20,1e,0) at kern_select+0x655 sys_select(cc0305c0,f3fe7cec,c1025d9c,c0fdb32e,c119b640,...) at sys_select+0x66 syscall(f3fe7d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28173743, esp = 0xbfbfeb0c, ebp = 0xbfbfec48 --- Tracing command nfsuserd pid 1027 tid 100094 td 0xcc0b3b80 sched_switch(cc0b3b80,0,104,1a3,7f6558db,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0b3b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f400ba14,c0a747fa,cbfa5090,0,cc0b3b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cbfa50a4,0,f400ba50,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cbfa50a4,cbfa5090,7531,6ae,cc05f1f8,...) at _cv_timedwait_sig+0x270 seltdwait(f400bc08,f400bc10,c95d9b80,cc0b3b80,f400bac4,...) at seltdwait+0x8a kern_select(cc0b3b80,4,bfbfebb4,0,0,f400bc54,20,1e,0) at kern_select+0x655 sys_select(cc0b3b80,f400bcec,c1025d9c,c0fdb32e,c119b640,...) at sys_select+0x66 syscall(f400bd28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28173743, esp = 0xbfbfeb0c, ebp = 0xbfbfec48 --- Tracing command nfsuserd pid 1026 tid 100093 td 0xcc0b4000 Tracing command mountd pid 1010 tid 100120 td 0xcae262e0 sched_switch(cae262e0,0,104,1a3,1468690e,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae262e0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc06edd0,0,c0fd08e7,cae262e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc06ede4,0,f40cea50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc06ede4,cc06edd0,c0fdaa18,6ae,cc03d620,...) at _cv_wait_sig+0x260 seltdwait(cc03d620,58,c95d9b80,cae262e0,f40ceac0,...) at seltdwait+0xa2 kern_select(cae262e0,9,bfbfecf4,0,0,0,20,cc0d0864,bfbfecf4) at kern_select+0x655 sys_select(cae262e0,f40cecec,c1025d9c,c0fbbf37,1,...) at sys_select+0x66 syscall(f40ced28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2818b743, esp = 0xbfbfec8c, ebp = 0xbfbfedd8 --- Tracing command rpcbind pid 976 tid 100112 td 0xcbee6b80 sched_switch(cbee6b80,0,104,1a3,eeaceb11,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cbee6b80,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f40b6a78,c0a747fa,cc0821d0,0,cbee6b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cc0821e4,0,f40b6ab4,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cc0821e4,cc0821d0,7531,6ae,f40b6b6c,...) at _cv_timedwait_sig+0x270 seltdwait(f40b6c3c,f40b6c44,5ab,cbee6b80,0,...) at seltdwait+0x8a sys_poll(cbee6b80,f40b6cec,c1025d9c,c0fdb32e,c1027350,...) at sys_poll+0x321 syscall(f40b6d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2812a0eb, esp = 0xbfbfcb9c, ebp = 0xbfbfed68 --- Tracing command syslogd pid 961 tid 100085 td 0xcc0308a0 sched_switch(cc0308a0,0,104,1a3,df47dde3,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0308a0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cbfa5dd0,0,c0fd08e7,cc0308a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cbfa5de4,0,f3fe3a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cbfa5de4,cbfa5dd0,c0fdaa18,6ae,cc03dc78,...) at _cv_wait_sig+0x260 seltdwait(cc03dc78,58,c95d9b80,cc0308a0,cae265c0,...) at seltdwait+0xa2 kern_select(cc0308a0,9,883d100,0,0,0,20,cc02b864,0) at kern_select+0x655 sys_select(cc0308a0,f3fe3cec,c1025d9c,c1008b8c,e,...) at sys_select+0x66 syscall(f3fe3d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2818b743, esp = 0xbfbfe28c, ebp = 0xbfbfedd8 --- Tracing command devd pid 838 tid 100096 td 0xcc0b35c0 sched_switch(cc0b35c0,0,104,1a3,12ff99a7,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cc0b35c0,0,c0fd82c0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a747fa,cc06ec50,0,c0fd08e7,cc0b35c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cc06ec64,0,f4013a50,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cc06ec64,cc06ec50,c0fdaa18,6ae,cc03d460,...) at _cv_wait_sig+0x260 seltdwait(cc03d460,58,c95d9b80,cc0b35c0,c1108c08,...) at seltdwait+0xa2 kern_select(cc0b35c0,5,bfbfe950,0,0,0,20,cc0ae2cc,46) at kern_select+0x655 sys_select(cc0b35c0,f4013cec,c1025d9c,c0e3242f,c119b640,...) at sys_select+0x66 syscall(f4013d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x808936b, esp = 0xbfbfe91c, ebp = 0xbfbfedf8 --- Tracing command softdepflush pid 19 tid 100078 td 0xcbee88a0 sched_switch(cbee88a0,0,104,1a3,a5ab49f9,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(cbee88a0,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c130c8c0,54,c100a3f1,0,0,...) at sleepq_timedwait+0x6b _sleep(c130c8c0,c130c880,54,c100a3f1,3e8,...) at _sleep+0x36a softdep_flush(0,efbe9d28,c0fcc8d0,3d8,cbedb000,...) at softdep_flush+0x259 fork_exit(c0cd7fd0,0,efbe9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbe9d60, ebp = 0 --- Tracing command vnlru pid 18 tid 100077 td 0xcbee8b80 sched_switch(cbee8b80,0,104,1a3,8f3ec6d3,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(cbee8b80,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(cbedb2cc,60,c0fe3415,0,0,...) at sleepq_timedwait+0x6b _sleep(cbedb2cc,c1304b74,260,c0fe3415,3e8,...) at _sleep+0x36a vnlru_proc(0,efbe6d28,c0fcc8d0,3d8,cbedb2cc,...) at vnlru_proc+0xe7 fork_exit(c0b2ba40,0,efbe6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbe6d60, ebp = 0 --- Tracing command syncer pid 17 tid 100076 td 0xcacf98a0 sched_switch(cacf98a0,0,104,1a3,65822fcb,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf98a0,0,c0fd82c0,28b,cacf98a0,...) at sleepq_switch+0x162 sleepq_timedwait(c1304bb4,0,efbe3c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c1304bb4,c1304ba0,3e8,783,4e20,...) at _cv_timedwait+0x270 sched_sync(0,efbe3d28,c0fcc8d0,3d8,cbedb598,...) at sched_sync+0x595 fork_exit(c0b2ad70,0,efbe3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbe3d60, ebp = 0 --- Tracing command bufdaemon pid 16 tid 100075 td 0xcacf9b80 sched_switch(cacf9b80,0,104,1a3,5d3f6730,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(cacf9b80,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1304848,54,c0fe018e,0,0,...) at sleepq_timedwait+0x6b _sleep(c1304848,c1304850,54,c0fe018e,3e8,...) at _sleep+0x36a buf_daemon(0,efbe0d28,c0fcc8d0,3d8,cbedb864,...) at buf_daemon+0x175 fork_exit(c0b0da90,0,efbe0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbe0d60, ebp = 0 --- Tracing command pagezero pid 9 tid 100074 td 0xcae24000 sched_switch(cae24000,0,104,1a3,8e494b32,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae24000,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c130e76c,0,c10115cd,0,0,...) at sleepq_timedwait+0x6b _sleep(c130e76c,c130d200,0,c10115cd,493e0,...) at _sleep+0x36a vm_pagezero(0,efbddd28,c0fcc8d0,3d8,cbedbb30,...) at vm_pagezero+0xdc fork_exit(c0d1fd40,0,efbddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbddd60, ebp = 0 --- Tracing command vmdaemon pid 8 tid 100073 td 0xcae242e0 sched_switch(cae242e0,0,104,1a3,fafd478a,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(cae242e0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c130e348,74,c0fe018e,0,0,...) at sleepq_wait+0x63 _sleep(c130e348,c130e34c,74,c0fe018e,0,...) at _sleep+0x39c vm_daemon(0,efbdad28,c0fcc8d0,3d8,cbedc000,...) at vm_daemon+0x59 fork_exit(c0d184d0,0,efbdad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbdad60, ebp = 0 --- Tracing command pagedaemon pid 7 tid 100072 td 0xcae245c0 sched_switch(cae245c0,0,104,1a3,8921451c,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(cae245c0,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c130e310,54,c0fe018e,0,0,...) at sleepq_timedwait+0x6b _sleep(c130e310,c130d200,54,c0fe018e,1388,...) at _sleep+0x36a vm_pageout(0,efbd7d28,c0fcc8d0,3d8,c95de2cc,...) at vm_pageout+0x2c7 fork_exit(c0d19550,0,efbd7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefbd7d60, ebp = 0 --- Tracing command xpt_thrd pid 6 tid 100071 td 0xcae248a0 sched_switch(cae248a0,0,104,1a3,c64e4980,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(cae248a0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1151354,5c,c0e2cc1e,0,0,...) at sleepq_wait+0x63 _sleep(c1151354,c1151398,5c,c0e2cc1e,0,...) at _sleep+0x39c xpt_scanner_thread(0,efa54d28,c0fcc8d0,3d8,c95de598,...) at xpt_scanner_thread+0x47 fork_exit(c04a72f0,0,efa54d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa54d60, ebp = 0 --- Tracing command sctp_iterator pid 5 tid 100070 td 0xcae24b80 sched_switch(cae24b80,0,104,1a3,5ea91afb,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae24b80,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c13080d8,0,c0ff78df,0,0,...) at sleepq_wait+0x63 _sleep(c13080d8,c13080a0,0,c0ff78df,0,...) at _sleep+0x39c sctp_iterator_thread(0,efa51d28,c0fcc8d0,3d8,c95de864,...) at sctp_iterator_thread+0x5c fork_exit(c0bc5c00,0,efa51d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa51d60, ebp = 0 --- Tracing command fdc0 pid 4 tid 100065 td 0xcae25b80 sched_switch(cae25b80,0,104,1a3,949bf93a,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(cae25b80,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(cac7223c,5c,c0fbec52,0,0,...) at sleepq_timedwait+0x6b _sleep(cac7223c,cac722f0,5c,c0fbec52,3e8,...) at _sleep+0x36a fdc_thread(cac72200,efa42d28,c0fcc8d0,3d8,c95deb30,...) at fdc_thread+0x27d fork_exit(c0da2080,cac72200,efa42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa42d60, ebp = 0 --- Tracing command fw0_probe pid 3 tid 100063 td 0xcace42e0 sched_switch(cace42e0,0,104,1a3,c64e871c,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(cace42e0,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cad2c000,6c,c0fbec52,100,0,...) at sleepq_wait_sig+0x17 _sleep(cad2c000,cad30488,16c,c0fbec52,0,...) at _sleep+0x385 fw_bus_probe_thread(cad2c000,efa32d28,c0fcc8d0,3d8,cac0b000,...) at fw_bus_probe_thread+0xa08 fork_exit(c0701880,cad2c000,efa32d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa32d60, ebp = 0 --- Tracing command usb pid 15 tid 100059 td 0xcacf7000 sched_switch(cacf7000,0,104,1a3,b3fc3616,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf7000,0,c0fd82c0,268,cacf7000,...) at sleepq_switch+0x162 sleepq_wait(cacfed34,0,efa1bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacfed34,cacfee4c,c0fb02d1,74,cacfed3c,...) at _cv_wait+0x260 usb_process(cacfed2c,efa1bd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacfed2c,efa1bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa1bd60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xcacf72e0 sched_switch(cacf72e0,0,104,1a3,4d635364,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf72e0,0,c0fd82c0,268,cacf72e0,...) at sleepq_switch+0x162 sleepq_wait(cacfed04,0,efa18cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacfed04,cacfee4c,c0fb02d1,74,cacfed0c,...) at _cv_wait+0x260 usb_process(cacfecfc,efa18d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacfecfc,efa18d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa18d60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xcacf75c0 sched_switch(cacf75c0,0,104,1a3,b395b902,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf75c0,0,c0fd82c0,268,cacf75c0,...) at sleepq_switch+0x162 sleepq_wait(cacfecd4,0,efa15cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacfecd4,cacfee4c,c0fb02d1,74,cacfecdc,...) at _cv_wait+0x260 usb_process(cacfeccc,efa15d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacfeccc,efa15d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa15d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xcacf78a0 sched_switch(cacf78a0,0,104,1a3,b395a9bd,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf78a0,0,c0fd82c0,268,cacf78a0,...) at sleepq_switch+0x162 sleepq_wait(cacfeca4,0,efa12cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacfeca4,cacfee4c,c0fb02d1,74,cacfecac,...) at _cv_wait+0x260 usb_process(cacfec9c,efa12d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacfec9c,efa12d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa12d60, ebp = 0 --- Tracing command usb pid 15 tid 100054 td 0xcacf9000 sched_switch(cacf9000,0,104,1a3,b3954c86,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf9000,0,c0fd82c0,268,cacf9000,...) at sleepq_switch+0x162 sleepq_wait(cacefb5c,0,efa09cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacefb5c,cacefc74,c0fb02d1,74,cacefb64,...) at _cv_wait+0x260 usb_process(cacefb54,efa09d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacefb54,efa09d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa09d60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xcacf92e0 sched_switch(cacf92e0,0,104,1a3,4d621e7d,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf92e0,0,c0fd82c0,268,cacf92e0,...) at sleepq_switch+0x162 sleepq_wait(cacefb2c,0,efa06cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacefb2c,cacefc74,c0fb02d1,74,cacefb34,...) at _cv_wait+0x260 usb_process(cacefb24,efa06d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacefb24,efa06d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa06d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xcacf95c0 sched_switch(cacf95c0,0,104,1a3,b330dbd2,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacf95c0,0,c0fd82c0,268,cacf95c0,...) at sleepq_switch+0x162 sleepq_wait(cacefafc,0,efa03cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacefafc,cacefc74,c0fb02d1,74,cacefb04,...) at _cv_wait+0x260 usb_process(cacefaf4,efa03d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacefaf4,efa03d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa03d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xcace2000 sched_switch(cace2000,0,104,1a3,b330c8a4,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace2000,0,c0fd82c0,268,cace2000,...) at sleepq_switch+0x162 sleepq_wait(cacefacc,0,efa00cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacefacc,cacefc74,c0fb02d1,74,cacefad4,...) at _cv_wait+0x260 usb_process(cacefac4,efa00d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacefac4,efa00d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa00d60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xcace22e0 sched_switch(cace22e0,0,104,1a3,b330ae7c,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace22e0,0,c0fd82c0,268,cace22e0,...) at sleepq_switch+0x162 sleepq_wait(caceab5c,0,ef9fccac,1,0,...) at sleepq_wait+0x63 _cv_wait(caceab5c,caceac74,c0fb02d1,74,caceab64,...) at _cv_wait+0x260 usb_process(caceab54,ef9fcd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,caceab54,ef9fcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9fcd60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xcace25c0 sched_switch(cace25c0,0,104,1a3,4d619267,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace25c0,0,c0fd82c0,268,cace25c0,...) at sleepq_switch+0x162 sleepq_wait(caceab2c,0,ef9f9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(caceab2c,caceac74,c0fb02d1,74,caceab34,...) at _cv_wait+0x260 usb_process(caceab24,ef9f9d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,caceab24,ef9f9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9f9d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xcace28a0 sched_switch(cace28a0,0,104,1a3,ad6fb3f1,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace28a0,0,c0fd82c0,268,cace28a0,...) at sleepq_switch+0x162 sleepq_wait(caceaafc,0,ef9f6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(caceaafc,caceac74,c0fb02d1,74,caceab04,...) at _cv_wait+0x260 usb_process(caceaaf4,ef9f6d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,caceaaf4,ef9f6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9f6d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xcace2b80 sched_switch(cace2b80,0,104,1a3,ad6fa257,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace2b80,0,c0fd82c0,268,cace2b80,...) at sleepq_switch+0x162 sleepq_wait(caceaacc,0,ef9f3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(caceaacc,caceac74,c0fb02d1,74,caceaad4,...) at _cv_wait+0x260 usb_process(caceaac4,ef9f3d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,caceaac4,ef9f3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9f3d60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xcace3000 sched_switch(cace3000,0,104,1a3,ad6f8dcf,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace3000,0,c0fd82c0,268,cace3000,...) at sleepq_switch+0x162 sleepq_wait(cace5b5c,0,ef9efcac,1,0,...) at sleepq_wait+0x63 _cv_wait(cace5b5c,cace5c74,c0fb02d1,74,cace5b64,...) at _cv_wait+0x260 usb_process(cace5b54,ef9efd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cace5b54,ef9efd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9efd60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xcace32e0 sched_switch(cace32e0,0,104,1a3,4d6104d1,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace32e0,0,c0fd82c0,268,cace32e0,...) at sleepq_switch+0x162 sleepq_wait(cace5b2c,0,ef9eccac,1,0,...) at sleepq_wait+0x63 _cv_wait(cace5b2c,cace5c74,c0fb02d1,74,cace5b34,...) at _cv_wait+0x260 usb_process(cace5b24,ef9ecd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cace5b24,ef9ecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9ecd60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xcace35c0 sched_switch(cace35c0,0,104,1a3,ad0b3f3f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace35c0,0,c0fd82c0,268,cace35c0,...) at sleepq_switch+0x162 sleepq_wait(cace5afc,0,ef9e9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cace5afc,cace5c74,c0fb02d1,74,cace5b04,...) at _cv_wait+0x260 usb_process(cace5af4,ef9e9d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cace5af4,ef9e9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9e9d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xcace38a0 sched_switch(cace38a0,0,104,1a3,ad0b2e30,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace38a0,0,c0fd82c0,268,cace38a0,...) at sleepq_switch+0x162 sleepq_wait(cace5acc,0,ef9e6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cace5acc,cace5c74,c0fb02d1,74,cace5ad4,...) at _cv_wait+0x260 usb_process(cace5ac4,ef9e6d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cace5ac4,ef9e6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9e6d60, ebp = 0 --- Tracing command usb pid 15 tid 100041 td 0xcace4000 sched_switch(cace4000,0,104,1a3,ad0b184f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace4000,0,c0fd82c0,268,cace4000,...) at sleepq_switch+0x162 sleepq_wait(cacddb5c,0,ef9dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacddb5c,cacddc74,c0fb02d1,74,cacddb64,...) at _cv_wait+0x260 usb_process(cacddb54,ef9dfd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacddb54,ef9dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9dfd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xcacbf000 sched_switch(cacbf000,0,104,1a3,4d6069ee,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacbf000,0,c0fd82c0,268,cacbf000,...) at sleepq_switch+0x162 sleepq_wait(cacddb2c,0,ef9dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacddb2c,cacddc74,c0fb02d1,74,cacddb34,...) at _cv_wait+0x260 usb_process(cacddb24,ef9dcd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacddb24,ef9dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9dcd60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xcacbf2e0 sched_switch(cacbf2e0,0,104,1a3,aca72849,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacbf2e0,0,c0fd82c0,268,cacbf2e0,...) at sleepq_switch+0x162 sleepq_wait(cacddafc,0,ef9d9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacddafc,cacddc74,c0fb02d1,74,cacddb04,...) at _cv_wait+0x260 usb_process(cacddaf4,ef9d9d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacddaf4,ef9d9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9d9d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xcacbf5c0 sched_switch(cacbf5c0,0,104,1a3,aca715b8,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacbf5c0,0,c0fd82c0,268,cacbf5c0,...) at sleepq_switch+0x162 sleepq_wait(cacddacc,0,ef9d6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacddacc,cacddc74,c0fb02d1,74,cacddad4,...) at _cv_wait+0x260 usb_process(cacddac4,ef9d6d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacddac4,ef9d6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9d6d60, ebp = 0 --- Tracing command usb pid 15 tid 100036 td 0xcacbfb80 sched_switch(cacbfb80,0,104,1a3,aca700bf,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacbfb80,0,c0fd82c0,268,cacbfb80,...) at sleepq_switch+0x162 sleepq_wait(cacd5b5c,0,ef9cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacd5b5c,cacd5c74,c0fb02d1,74,cacd5b64,...) at _cv_wait+0x260 usb_process(cacd5b54,ef9cfd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacd5b54,ef9cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9cfd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xcacc1000 sched_switch(cacc1000,0,104,1a3,4d4486ea,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacc1000,0,c0fd82c0,268,cacc1000,...) at sleepq_switch+0x162 sleepq_wait(cacd5b2c,0,ef9cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacd5b2c,cacd5c74,c0fb02d1,74,cacd5b34,...) at _cv_wait+0x260 usb_process(cacd5b24,ef9ccd28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacd5b24,ef9ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9ccd60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xcacc12e0 sched_switch(cacc12e0,0,104,1a3,ac430ee0,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacc12e0,0,c0fd82c0,268,cacc12e0,...) at sleepq_switch+0x162 sleepq_wait(cacd5afc,0,ef9c9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacd5afc,cacd5c74,c0fb02d1,74,cacd5b04,...) at _cv_wait+0x260 usb_process(cacd5af4,ef9c9d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacd5af4,ef9c9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9c9d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xcacc15c0 sched_switch(cacc15c0,0,104,1a3,ac42fa90,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cacc15c0,0,c0fd82c0,268,cacc15c0,...) at sleepq_switch+0x162 sleepq_wait(cacd5acc,0,ef9c6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(cacd5acc,cacd5c74,c0fb02d1,74,cacd5ad4,...) at _cv_wait+0x260 usb_process(cacd5ac4,ef9c6d28,c0fcc8d0,3d8,cac0b2cc,...) at usb_process+0x1a3 fork_exit(c092f860,cacd5ac4,ef9c6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9c6d60, ebp = 0 --- Tracing command ctl_thrd pid 2 tid 100028 td 0xc96eb8a0 sched_switch(c96eb8a0,0,104,1a3,ac42df1b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c96eb8a0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c97c4000,5c,c0e37690,0,0,...) at sleepq_wait+0x63 _sleep(c97c4000,c97c4000,5c,c0e37690,0,...) at _sleep+0x39c ctl_work_thread(c97c4000,ef733d28,c0fcc8d0,3d8,cac0b598,...) at ctl_work_thread+0x22f6 fork_exit(c04d08d0,c97c4000,ef733d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef733d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc96ea5c0 sched_switch(c96ea5c0,0,104,1a3,beaac09f,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c96ea5c0,0,c0fd82c0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c118d7e4,0,c0fbec52,2,0,...) at sleepq_timedwait+0x6b _sleep(c118d7e4,0,0,c0fbec52,64) at _sleep+0x36a pause(c0fbec52,64,c0f8c330,111,0,...) at pause+0xa0 random_kthread(0,e33e0d28,c0fcc8d0,3d8,c95dd000,...) at random_kthread+0x1ef fork_exit(c082d0c0,0,e33e0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33e0d60, ebp = 0 --- Tracing command geom pid 13 tid 100016 td 0xc96ea8a0 sched_switch(c96ea8a0,0,104,1a3,e6832738,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c96ea8a0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c118a8a8,5c,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c118a8a8,c118a808,25c,c0fbec52,0,...) at _sleep+0x39c g_io_schedule_down(c96ea8a0,0,c0fc07a1,6c,e33ddd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,e33ddd28,c0fcc8d0,3d8,c95dd2cc,...) at g_down_procbody+0x9d fork_exit(c09e6b60,0,e33ddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33ddd60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc96eab80 sched_switch(c96eab80,0,104,1a3,fef34f71,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c96eab80,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c118a8a4,5c,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c118a8a4,c118a828,25c,c0fbec52,0,...) at _sleep+0x39c g_io_schedule_up(c96eab80,0,c0fc07a1,5f,e33dad14,...) at g_io_schedule_up+0x134 g_up_procbody(0,e33dad28,c0fcc8d0,3d8,c95dd2cc,...) at g_up_procbody+0x9d fork_exit(c09e6c00,0,e33dad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33dad60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc96eb000 sched_switch(c96eb000,0,104,1a3,12fe7765,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c96eb000,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c118a89c,5c,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c118a89c,c118a7cc,25c,c0fbec52,0,...) at _sleep+0x39c g_run_events(c119b640,0,c0fc07a1,79,e33d7d14,...) at g_run_events+0x54b g_event_procbody(0,e33d7d28,c0fcc8d0,3d8,c95dd2cc,...) at g_event_procbody+0x90 fork_exit(c09e6ca0,0,e33d7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33d7d60, ebp = 0 --- Tracing command intr pid 12 tid 100067 td 0xcae255c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100066 td 0xcae258a0 sched_switch(cae258a0,0,109,1a3,ac3deff3,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cae258a0,...) at mi_switch+0x1ff ithread_loop(cae1e990,efa45d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cae1e990,efa45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa45d60, ebp = 0 --- Tracing command intr pid 12 tid 100064 td 0xcae26000 sched_switch(cae26000,0,109,1a3,97b290ed,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cae26000,...) at mi_switch+0x1ff ithread_loop(cae18a40,efa35d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cae18a40,efa35d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa35d60, ebp = 0 --- Tracing command intr pid 12 tid 100061 td 0xcace48a0 sched_switch(cace48a0,0,109,1a3,9cb02074,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cace48a0,...) at mi_switch+0x1ff ithread_loop(cad15880,efa29d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cad15880,efa29d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa29d60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xcace4b80 sched_switch(cace4b80,0,109,1a3,c52ce77f,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cace4b80,...) at mi_switch+0x1ff ithread_loop(cad15010,efa21d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cad15010,efa21d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa21d60, ebp = 0 --- Tracing command intr pid 12 tid 100055 td 0xcacf7b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100042 td 0xcace3b80 sched_switch(cace3b80,0,109,1a3,d0d85354,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cace3b80,...) at mi_switch+0x1ff ithread_loop(cacc5720,ef9e3d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cacc5720,ef9e3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9e3d60, ebp = 0 --- Tracing command intr pid 12 tid 100037 td 0xcacbf8a0 sched_switch(cacbf8a0,0,109,1a3,cbcd8c13,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cacbf8a0,...) at mi_switch+0x1ff ithread_loop(cacad590,ef9d3d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cacad590,ef9d3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9d3d60, ebp = 0 --- Tracing command intr pid 12 tid 100032 td 0xcacc18a0 sched_switch(cacc18a0,0,109,1a3,3314795c,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cacc18a0,...) at mi_switch+0x1ff ithread_loop(cacad830,ef9c3d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cacad830,ef9c3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef9c3d60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xcacc1b80 sched_switch(cacc1b80,0,109,1a3,ba0de7a6,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,cacc1b80,...) at mi_switch+0x1ff ithread_loop(cac97ae0,ef86dd28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cac97ae0,ef86dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef86dd60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc96eb2e0 sched_switch(c96eb2e0,0,109,1a3,a523fda1,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c96eb2e0,...) at mi_switch+0x1ff ithread_loop(cac97c80,ef869d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cac97c80,ef869d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef869d60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc96eb5c0 sched_switch(c96eb5c0,0,109,1a3,9e897243,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c96eb5c0,...) at mi_switch+0x1ff ithread_loop(cac97500,ef763d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,cac97500,ef763d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef763d60, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc97c1b80 sched_switch(c97c1b80,0,109,1a3,56cb3808,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c97c1b80,...) at mi_switch+0x1ff ithread_loop(c95b3450,e33efd28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3450,e33efd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33efd60, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc96228a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xc96ea000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100018 td 0xc96ea2e0 sched_switch(c96ea2e0,0,109,1a3,4899bacf,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c96ea2e0,...) at mi_switch+0x1ff ithread_loop(c95b3490,e33e3d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3490,e33e3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33e3d60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc95e05c0 sched_switch(c95e05c0,0,109,1a3,c536307f,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c95e05c0,...) at mi_switch+0x1ff ithread_loop(c95b36f0,e33d1d28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b36f0,e33d1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33d1d60, ebp = 0 --- Tracing command intr pid 12 tid 100011 td 0xc95e08a0 sched_switch(c95e08a0,0,109,1a3,bb8e08fd,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c95e08a0,...) at mi_switch+0x1ff ithread_loop(c95b3700,e33ced28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3700,e33ced28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33ced60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc95e0b80 sched_switch(c95e0b80,0,109,1a3,bb04f564,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c95e0b80,...) at mi_switch+0x1ff ithread_loop(c95b3710,e33cbd28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3710,e33cbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33cbd60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc9622000 cpustop_handler(0,c93febc8,c0e00e13,cace45c0,c95be180,...) at cpustop_handler+0x26 ipi_nmi_handler(cace45c0,c95be180,c93feb14,c0a905a8,c95dd598,...) at ipi_nmi_handler+0x35 trap(c93febd4) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0a74276, esp = 0xc93fec14, ebp = 0xc93fec30 --- _mtx_unlock_spin_flags(c119b640,0,c0fd08e7,294,c96222b8,...) at _mtx_unlock_spin_flags+0x46 thread_lock_block(c9622000,0,c0fd494c,747,c0a758eb,...) at thread_lock_block+0x60 sched_switch(c9622000,0,109,1a3,c5368a14,...) at sched_switch+0x36e mi_switch(109,0,c0fccc01,557,c9622000,...) at mi_switch+0x1ff ithread_loop(c95b3720,c93fed28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3720,c93fed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93fed60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc96222e0 sched_switch(c96222e0,0,109,1a3,a0f07ad6,...) at sched_switch+0x413 mi_switch(109,0,c0fccc01,557,c96222e0,...) at mi_switch+0x1ff ithread_loop(c95b3730,c93fbd28,c0fcc8d0,3d8,c95dd598,...) at ithread_loop+0x1fb fork_exit(c0a5aa30,c95b3730,c93fbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93fbd60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc96225c0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xc95df000 cpustop_handler(3,c93f4bac,c0e00e13,c93f4b30,c136d680,...) at cpustop_handler+0x26 ipi_nmi_handler(c93f4b30,c136d680,c93f4b00,c93f4b00,c95dd864,...) at ipi_nmi_handler+0x35 trap(c93f4bb8) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc05293d6, esp = 0xc93f4bf8, ebp = 0xc93f4bf8 --- AcpiOsReadPort(808,c93f4c64,20,0,c1151930,...) at AcpiOsReadPort+0x36 AcpiHwReadPort(808,c93f4c64,20,4,c0fd08e7,...) at AcpiHwReadPort+0x3e AcpiHwRead(c93f4c64,c1151930,3,317,3fe5,...) at AcpiHwRead+0x77 acpi_cpu_idle(0,c93f4c98,c0df3719,0,3,...) at acpi_cpu_idle+0x11d cpu_idle_acpi(0,3,0,2,c93f4ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c93f4cc0,c0fd494c,a1f,c95df000,...) at cpu_idle+0x89 sched_idletd(0,c93f4d28,c0fcc8d0,3d8,c95dd864,...) at sched_idletd+0x235 fork_exit(c0aaf1f0,0,c93f4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93f4d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc95df2e0 cpustop_handler(2,c93f1bfc,c0e00e13,c93dec80,c93f1b4c,...) at cpustop_handler+0x26 ipi_nmi_handler(c93dec80,c93f1b4c,c0a74363,c93dec80,c95dd864,...) at ipi_nmi_handler+0x35 trap(c93f1c08) at trap+0x33 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0de64b5, esp = 0xc93f1c48, ebp = 0xc93f1c48 --- acpi_cpu_c1(c93f1c64,c1151930,2,317,3fe5,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c93f1c98,c0df3719,0,2,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,2,0,2,c93f1ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c93f1cc0,c0fd494c,a1f,c95df2e0,...) at cpu_idle+0x89 sched_idletd(0,c93f1d28,c0fcc8d0,3d8,c95dd864,...) at sched_idletd+0x235 fork_exit(c0aaf1f0,0,c93f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93f1d60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc95df5c0 sched_switch(c95df5c0,0,608,19e,c5367859,...) at sched_switch+0x413 mi_switch(608,0,c0fd2f7f,cd,0,...) at mi_switch+0x1ff critical_exit(0,1,0,2,c93eece8,...) at critical_exit+0xb5 cpu_idle(0,c93eecc0,c0fd494c,a1f,c95df5c0,...) at cpu_idle+0x97 sched_idletd(0,c93eed28,c0fcc8d0,3d8,c95dd864,...) at sched_idletd+0x235 fork_exit(c0aaf1f0,0,c93eed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93eed60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc95df8a0 sched_switch(c95df8a0,0,608,19e,c53596e6,...) at sched_switch+0x413 mi_switch(608,0,c0fd2f7f,cd,0,...) at mi_switch+0x1ff critical_exit(0,0,0,2,c93ebce8,...) at critical_exit+0xb5 cpu_idle(0,c93ebcc0,c0fd494c,a1f,c95df8a0,...) at cpu_idle+0x97 sched_idletd(0,c93ebd28,c0fcc8d0,3d8,c95dd864,...) at sched_idletd+0x235 fork_exit(c0aaf1f0,0,c93ebd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93ebd60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc95dfb80 sched_switch(c95dfb80,0,104,1a3,7799b6d,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c95dfb80,0,c0fd82c0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0fd82c0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c95ddb30,6c,c0fdb1f8,100,0,...) at sleepq_wait_sig+0x17 _sleep(c95ddb30,c95ddbb8,16c,c0fdb1f8,0,...) at _sleep+0x385 kern_wait(c95dfb80,ffffffff,c93e7c58,0,0,...) at kern_wait+0x597 sys_wait4(c95dfb80,c93e7cec,c1025d9c,c0fdb046,d5ded2e0,...) at sys_wait4+0x3b syscall(c93e7d28) at syscall+0x2fe Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x806070f, esp = 0xbfbfe8ac, ebp = 0xbfbfe8c8 --- Tracing command audit pid 10 tid 100001 td 0xc95e0000 sched_switch(c95e0000,0,104,1a3,ac40a142,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c95e0000,0,c0fd82c0,268,c95e0000,...) at sleepq_switch+0x162 sleepq_wait(c130c000,0,c93e4c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c130c000,c130bfe4,c1004577,194,c93e4cc4,...) at _cv_wait+0x260 audit_worker(0,c93e4d28,c0fcc8d0,3d8,c95de000,...) at audit_worker+0x84 fork_exit(c0c952c0,0,c93e4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc93e4d60, ebp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xcae25000 sched_switch(cae25000,0,104,1a3,8eb6de38,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae25000,0,c0fd82c0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c118d7e4,0,c0fbec52,2,0,...) at sleepq_timedwait+0x6b _sleep(c118d7e4,0,0,c0fbec52,bb8) at _sleep+0x36a pause(c0fbec52,bb8,126,124,c118aae8,...) at pause+0xa0 deadlkres(0,efa4ed28,c0fcc8d0,3d8,c118aa60,...) at deadlkres+0x324 fork_exit(c0a39910,0,efa4ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa4ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xcae252e0 sched_switch(cae252e0,0,104,1a3,66fc27cd,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cae252e0,0,c0fd82c0,268,cae252e0,...) at sleepq_switch+0x162 sleepq_wait(c971ddc0,0,c0fd34a1,c0fbec52,0,...) at sleepq_wait+0x63 msleep_spin(c971ddc0,c971ddd8,c0fbec52,0,a55,...) at msleep_spin+0x25f taskqueue_thread_loop(c136d740,efa4bd28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0x8f fork_exit(c0acc2b0,c136d740,efa4bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa4bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100062 td 0xcace45c0 sched_switch(cace45c0,0,104,1a3,c4ffa732,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(cace45c0,0,c0fd82c0,268,cace45c0,...) at sleepq_switch+0x162 sleepq_wait(c971de00,0,c0fd34a1,c0fbec52,0,...) at sleepq_wait+0x63 msleep_spin(c971de00,c971de18,c0fbec52,0,a55,...) at msleep_spin+0x25f taskqueue_thread_loop(cad3049c,efa2fd28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0x8f fork_exit(c0acc2b0,cad3049c,efa2fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xefa2fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc96ebb80 sched_switch(c96ebb80,0,104,1a3,c652de07,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c96ebb80,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c971de40,0,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c971de40,c971de58,0,c0fbec52,0,...) at _sleep+0x39c taskqueue_thread_loop(c118b3f8,e33fed28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acc2b0,c118b3f8,e33fed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33fed60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc97c1000 sched_switch(c97c1000,0,104,1a3,8c805014,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c97c1000,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c971de80,0,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c971de80,c971de98,0,c0fbec52,0,...) at _sleep+0x39c taskqueue_thread_loop(c130c5e4,e33fbd28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acc2b0,c130c5e4,e33fbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33fbd60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc97c12e0 sched_switch(c97c12e0,0,104,1a3,c652b987,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c97c12e0,0,c0fd82c0,268,c97c12e0,...) at sleepq_switch+0x162 sleepq_wait(c971e000,0,c0fd34a1,c0fbec52,0,...) at sleepq_wait+0x63 msleep_spin(c971e000,c971e018,c0fbec52,0,a55,...) at msleep_spin+0x25f taskqueue_thread_loop(c1154598,e33f8d28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0x8f fork_exit(c0acc2b0,c1154598,e33f8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33f8d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc97c15c0 sched_switch(c97c15c0,0,104,1a3,c652a818,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c97c15c0,0,c0fd82c0,268,c97c15c0,...) at sleepq_switch+0x162 sleepq_wait(c971e000,0,c0fd34a1,c0fbec52,0,...) at sleepq_wait+0x63 msleep_spin(c971e000,c971e018,c0fbec52,0,a55,...) at msleep_spin+0x25f taskqueue_thread_loop(c1154598,e33f5d28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0x8f fork_exit(c0acc2b0,c1154598,e33f5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33f5d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc97c18a0 sched_switch(c97c18a0,0,104,1a3,c652988b,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c97c18a0,0,c0fd82c0,268,c97c18a0,...) at sleepq_switch+0x162 sleepq_wait(c971e000,0,c0fd34a1,c0fbec52,0,...) at sleepq_wait+0x63 msleep_spin(c971e000,c971e018,c0fbec52,0,a55,...) at msleep_spin+0x25f taskqueue_thread_loop(c1154598,e33f2d28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0x8f fork_exit(c0acc2b0,c1154598,e33f2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33f2d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc9622b80 sched_switch(c9622b80,0,104,1a3,a285ed92,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c9622b80,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c971e0c0,0,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c971e0c0,c971e0d8,0,c0fbec52,0,...) at _sleep+0x39c taskqueue_thread_loop(c11aa3ac,e33e9d28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acc2b0,c11aa3ac,e33e9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33e9d60, ebp = 0 --- Tracing command kernel pid 0 tid 100013 td 0xc95e02e0 sched_switch(c95e02e0,0,104,1a3,bf5ac72a,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c95e02e0,0,c0fd82c0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c96e9000,0,c0fbec52,0,0,...) at sleepq_wait+0x63 _sleep(c96e9000,c96e9018,0,c0fbec52,0,...) at _sleep+0x39c taskqueue_thread_loop(c11a8dc0,e33d4d28,c0fcc8d0,3d8,c118aa60,...) at taskqueue_thread_loop+0xb5 fork_exit(c0acc2b0,c11a8dc0,e33d4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe33d4d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc118ad30 sched_switch(c118ad30,0,104,1a3,8da49f70,...) at sched_switch+0x413 mi_switch(104,0,c0fd82c0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c118ad30,0,c0fd82c0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c118aa60,54,c1004f58,0,0,...) at sleepq_timedwait+0x6b _sleep(c118aa60,0,54,c1004f58,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Dumping 555 MB: 540 524 508 492 476 460 444 428 412 396 380 364 348 332 316 300 284 268 252 236 220 204 188 172 156 140 124 108 92 76 60 44 28 12 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump (textdump=0x0) at pcpu.h:244 #1 0xc051b3a9 in db_fncall (dummy1=0xc0abd9ba, dummy2=0x0, dummy3=0xffffffff, dummy4=0xf4164210 "$B\026ô") at ../../../ddb/db_command.c:573 #2 0xc051b7df in db_command (last_cmdp=0xc115275c, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:449 #3 0xc051b894 in db_command_script (command=0xc1153668 "call doadump") at ../../../ddb/db_command.c:520 #4 0xc051fcd0 in db_script_exec (scriptname=0xc1152fc0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc051fd61 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc132dd60, modif=0xf4164348 "") at ../../../ddb/db_script.c:375 #6 0xc051b7a1 in db_command (last_cmdp=0xc115275c, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:449 #7 0xc051b8fa in db_command_loop () at ../../../ddb/db_command.c:502 #8 0xc051d8ed in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:231 #9 0xc0abded6 in kdb_trap (type=0x3, code=0x0, tf=0xf4164544) at ../../../kern/subr_kdb.c:654 #10 0xc0e0157b in trap (frame=0xf4164544) at ../../../i386/i386/trap.c:710 #11 0xc0deab4c in calltrap () at ../../../i386/i386/exception.s:169 #12 0xc0abd9ba in kdb_enter (why=0xc0fd2704 "panic", msg=0xc0fd2704 "panic") at cpufunc.h:71 #13 0xc0a86eac in panic (fmt=0xc100e2a4 "kmem_malloc(%ld): kmem_map too small: %ld total allocated") at ../../../kern/kern_shutdown.c:633 #14 0xc0d073c0 in kmem_malloc (map=0xc1bb5118, size=0x20000, flags=0x2) at ../../../vm/vm_kern.c:304 #15 0xc0cfc5c7 in page_alloc (zone=0x0, bytes=0x20000, pflag=0xf416461b "\002", wait=0x2) at ../../../vm/uma_core.c:1002 #16 0xc0cfe290 in uma_large_malloc (size=0x20000, wait=0x2) at ../../../vm/uma_core.c:3106 #17 0xc0a712bd in malloc (size=0x20000, mtp=0xcc1c4130, flags=0x2) at ../../../kern/kern_malloc.c:529 #18 0xcc1b95c9 in gj_malloc (size=0x20000, flags=0x2) at /var/tmp/vmcontention/sys/modules/geom/geom_journal/../../../geom/journal/g_journal.c:278 #19 0xcc1b9702 in g_journal_new_bio (start=0x3af18000, end=Unhandled dwarf expression opcode 0x93 ) at /var/tmp/vmcontention/sys/modules/geom/geom_journal/../../../geom/journal/g_journal.c:778 #20 0xcc1b9820 in g_journal_insert (head=0xcc679cc0, nstart=0x3af18000, nend=0x3af38000, joffset=0x7bff6600, data=0xde19b000 "", flags=0x1) at /var/tmp/vmcontention/sys/modules/geom/geom_journal/../../../geom/journal/g_journal.c:836 #21 0xcc1bce9c in g_journal_flush (sc=0xcc679c00) at /var/tmp/vmcontention/sys/modules/geom/geom_journal/../../../geom/journal/g_journal.c:1321 #22 0xcc1c15f6 in g_journal_worker (arg=0xcc679c00) at /var/tmp/vmcontention/sys/modules/geom/geom_journal/../../../geom/journal/g_journal.c:1466 #23 0xc0a56ed8 in fork_exit (callout=0xcc1bff50 , arg=0xcc679c00, frame=0xf4164d28) at ../../../kern/kern_fork.c:992 #24 0xc0deabf4 in fork_trampoline () at ../../../i386/i386/exception.s:276 (kgdb) f 14 #14 0xc0d073c0 in kmem_malloc (map=0xc1bb5118, size=0x20000, flags=0x2) at ../../../vm/vm_kern.c:304 304 panic("kmem_malloc(%ld): kmem_map too small: %ld total allocated", (kgdb) l 299 } 300 vm_map_unlock(map); 301 tsleep(&i, 0, "nokva", (hz / 4) * (i + 1)); 302 } 303 if (i == 8) { 304 panic("kmem_malloc(%ld): kmem_map too small: %ld total allocated", 305 (long)size, (long)map->size); 306 } 307 } else { 308 return (0); (kgdb) $ svn diff -x -p /var/tmp/vmcontention/sys Index: /var/tmp/vmcontention/sys/vm/vm_radix.c =================================================================== --- /var/tmp/vmcontention/sys/vm/vm_radix.c (revision 235350) +++ /var/tmp/vmcontention/sys/vm/vm_radix.c (working copy) @@ -97,6 +97,7 @@ struct vm_radix_node { CTASSERT(sizeof(struct vm_radix_node) < PAGE_SIZE); static uma_zone_t vm_radix_node_zone; +static u_long allocfail; #ifndef UMA_MD_SMALL_ALLOC static vm_map_t rnode_map; @@ -192,6 +193,10 @@ static __inline struct vm_radix_node * vm_radix_node_get(void) { + if (++allocfail == 100000000) { + allocfail = 0; + return (NULL); + } return (uma_zalloc(vm_radix_node_zone, M_NOWAIT | M_ZERO)); } @@ -280,11 +285,11 @@ vm_radix_setroot(struct vm_radix *rtree, struct vm rtree->rt_root = root; } -static inline void -vm_radix_unwind_heightup(struct vm_radix *rtree, struct vm_radix_node *root, - struct vm_radix_node *iroot, int ilevel) +static void +vm_radix_unwind_heightup(struct vm_radix *rtree, volatile struct vm_radix_node *root, + volatile struct vm_radix_node *iroot, int ilevel) { - struct vm_radix_node *rnode; + volatile struct vm_radix_node *rnode; CTR4(KTR_VM, "unwind: tree %p, root %p, iroot %p, ilevel %d", rtree, root, iroot, ilevel); @@ -293,9 +298,9 @@ vm_radix_setroot(struct vm_radix *rtree, struct vm MPASS(rnode->rn_count == 0 || rnode->rn_count == 1); rnode->rn_count = 0; root = rnode->rn_child[0]; - vm_radix_node_put(rnode); + vm_radix_node_put(__DEVOLATILE(struct vm_radix_node *, rnode)); } - vm_radix_setroot(rtree, iroot, ilevel); + vm_radix_setroot(rtree, __DEVOLATILE(struct vm_radix_node *, iroot), ilevel); } static inline void * @@ -349,15 +354,15 @@ vm_radix_reclaim_allnodes_internal(struct vm_radix int vm_radix_insert(struct vm_radix *rtree, vm_pindex_t index, void *val) { - struct vm_radix_node *iroot, *rnode, *root; - u_int allocmsk; - int clev, ilevel, level, slot; + volatile struct vm_radix_node *iroot, *rnode, *root; + volatile u_int allocmsk; + volatile int clev, ilevel, level, slot; CTR3(KTR_VM, "insert: tree %p, index %ju, val %p", rtree, (uintmax_t)index, val); if (index == -1) panic("vm_radix_insert: -1 is not a valid index.\n"); - level = vm_radix_height(rtree, &root); + level = vm_radix_height(rtree, __DEVOLATILE(struct vm_radix_node **, &root)); /* * Increase the height by adding nodes at the root until * there is sufficient space. @@ -395,7 +400,7 @@ vm_radix_insert(struct vm_radix *rtree, vm_pindex_ } root = rnode; } - vm_radix_setroot(rtree, root, level); + vm_radix_setroot(rtree, __DEVOLATILE(struct vm_radix_node *, root), level); } /* Now that the tree is tall enough, fill in the path to the index. */ -- Test scenario: gjournal.sh Type InUse MemUse HighUse Requests Size(s) md_nvidia_data 0 0K - 51039 512 md_sii_data 0 0K - 51039 512 kbdmux 6 18K - 6 16,256,1024,2048 feeder 21 1K - 27 16,64 acpitask 1 1K - 1 1024 CAM periph 6 1K - 28 16,32,64,128 CAM XPT 89 94K - 286 16,32,64,1024,2048 isadev 7 1K - 7 64 acpisem 17 2K - 17 64,128 mixer 4 16K - 4 4096 md_disk 2 4K - 52 2048 scsi_cd 0 0K - 11 16 cdev 8 1K - 8 128 sigio 1 1K - 1 32 filedesc 84 28K - 1687941 16,32,256,512,1024,2048 kdtrace 717 166K - 5677164 64,256 kenv 86 8K - 107 16,32,64,128,4096 kqueue 2 2K - 40456 128,1024 proc-args 49 3K - 189363 16,32,64,128,256 hhook 2 1K - 2 128 ithread 88 7K - 88 16,64,128 CAM dev queue 7 1K - 7 128 KTRACE 100 13K - 100 128 linker 185 1133K - 267 16,32,512,1024,2048,4096 lockf 32 3K - 60026354 32,64,128,256 loginclass 2 1K - 317 64 ctlmem 5062 10113K - 5062 64,2048 ctlblk 200 800K - 200 4096 ip6ndp 12 1K - 13 64,128 temp 36 225K - 4515140 16,32,64,128,256,512,1024,2048,4096 devbuf 2858 26572K - 34717 16,32,64,128,256,512,1024,2048,4096 module 503 32K - 503 64,128 UART 3 2K - 3 16,512,1024 mtx_pool 2 8K - 2 4096 ramdisk 1 4096K - 1 osd 2 1K - 2 16,32 pmchooks 1 1K - 1 64 subproc 656 442K - 1676843 256,4096 proc 2 8K - 2 4096 session 32 2K - 144 64 pgrp 36 3K - 370 64 cred 136 13K - 5926898 64,128 uidinfo 4 2K - 114 64,1024 plimit 18 5K - 1549 256 acpidev 31 1K - 31 32 ctlpool 532 138K - 532 16,512 sysctltmp 0 0K - 1365 16,32,64,128,256,4096 sysctloid 3983 123K - 4109 16,32,64,128 sysctl 0 0K - 415823 16,32,64 tidhash 1 8K - 1 callout 3 768K - 3 umtx 1940 182K - 1980 64,128 p1003.1b 1 1K - 1 16 SWAP 2 2189K - 4 64 pci_link 16 2K - 16 64,128 bus-sc 86 150K - 3595 16,32,64,128,256,512,1024,2048,4096 bus 1232 54K - 74455 16,32,64,128,256,1024 devstat 30 61K - 30 16,4096 eventhandler 87 4K - 87 32,64,128 kobj 349 698K - 306776 2048 Per-cpu 1 1K - 1 16 USBdev 24 2K - 24 32,64,128 USB 42 8K - 42 16,32,64,1024 rman 229 14K - 642 16,32,64 CAM queue 25 1K - 138 16,128 sbuf 0 0K - 710909 16,32,64,128,256,512,1024,2048,4096 CAM SIM 7 1K - 7 128 stack 0 0K - 2 128 taskqueue 17 1K - 17 16,64 Unitno 31 2K - 4265890 16,64 ddb_capture 1 48K - 1 Witness 1 104K - 1 iov 0 0K - 551786 16,64,128,256,512 select 42 3K - 42 64 ioctlops 0 0K - 30988 16,32,64,128,256,512,1024 msg 4 25K - 4 1024,4096 sem 4 101K - 4 1024,4096 shm 1 12K - 1 tty 22 11K - 24 512,2048 pts 2 1K - 2 128 mbuf_tag 0 0K - 9 32 shmfd 1 4K - 1 4096 pcb 29 79K - 337 16,64,512,1024,2048,4096 soname 5 1K - 669474 16,32,128 acl 0 0K - 103878 4096 biobuf 0 0K - 53001 2048 vfscache 1 512K - 1 cl_savebuf 0 0K - 1725806 32,64,128 export_host 30 4K - 21103 128 vfs_hash 1 256K - 1 vnodes 2 1K - 212 32,128 fw_xfer 0 0K - 1 128 mount 148 12K - 3872693 16,32,64,128,256 vnodemarker 0 0K - 2008146 512 BPF 10 1K - 10 64 ether_multi 40 2K - 46 16,32,64 ifaddr 86 19K - 86 16,32,64,128,256,512,2048 ifnet 11 11K - 11 64,1024 firewire 11 23K - 14 32,64,512,1024,2048,4096 arpcom 2 1K - 2 16 fw_com 1 1K - 1 64 lltable 26 7K - 34 128,256 DEVFS1 140 35K - 17196 256 DEVFS3 163 21K - 17383 128 DEVFS2 136 3K - 139 16 DEVFS 34 1K - 67 16,64 routetbl 63 11K - 14621 16,32,64,128,256 igmp 10 2K - 10 128 in_multi 2 1K - 2 128 sctp_iter 0 0K - 3 256 sctp_ifn 2 1K - 2 128 sctp_ifa 5 1K - 5 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 3 16 hostcache 1 16K - 1 syncache 1 72K - 1 ip6_moptions 2 1K - 2 32,128 in6_multi 22 3K - 22 16,256 in6_mfilter 1 1K - 1 512 NFSCL diroffdiroff 0 0K - 92 512 mld 10 2K - 10 128 rpc 26 10K - 6039066 16,32,64,128,256,512,1024,2048,4096 audit_evclass 179 3K - 218 16 jblocks 0 0K - 18 128 savedino 0 0K - 17343109 256 sbdep 0 0K - 185377 32 jsegdep 0 0K - 43881959 32 jseg 0 0K - 546757 128 jfreefrag 0 0K - 1832938 64 jnewblk 0 0K - 7308625 64 jmvref 0 0K - 5355353 64 jremref 0 0K - 17181975 64 jaddref 0 0K - 17558421 64 freedep 0 0K - 6016 32 freework 2 1K - 3558700 16,128 newdirblk 0 0K - 2425948 32 dirrem 2 1K - 37266043 64 mkdir 0 0K - 4724456 64 diradd 1 1K - 37416700 64 freefile 3 1K - 11091407 32 freeblks 1 1K - 2531877 128 freefrag 0 0K - 3725969 64 indirdep 0 0K - 98389 64 newblk 2 65K - 12342790 128 bmsafemap 2 5K - 86731 128,4096 inodedep 7 258K - 13870511 256 pagedep 2 65K - 2874922 128 ufs_dirhash 287 144K - 553417 16,32,64,128,256,512,1024,2048,4096 ufs_quota 5 257K - 5 128 ufs_mount 24 89K - 50751 16,256,1024,2048,4096 vm_pgdata 2 129K - 3 64 UMAHash 2 5K - 8 256,512,1024,2048,4096 atkbddev 2 1K - 2 32 NFS fh 0 0K - 1794033 32 NFSD usrgroup 75 5K - 76 64 NFSD string 1 1K - 1 16 NFSD srvcache 502 32K - 2222065 64 newnfsmnt 0 0K - 4 512 newnfsclient_req 0 0K - 1 64 memdesc 1 4K - 1 4096 pfs_nodes 21 3K - 21 128 pfs_vncache 1 1K - 2593 32 GEOM 237 52K - 1397882 16,32,64,128,512,1024,2048 entropy 1024 64K - 1024 64 acpiintr 1 1K - 1 32 apmdev 1 1K - 1 64 acpica 2567 134K - 57306 16,32,64,128,256,512,1024 madt_table 0 0K - 1 2048 hdaa 9 44K - 9 256,512,1024,2048 hdac 2 1K - 2 64,512 hdacc 2 1K - 2 16 io_apic 1 1K - 1 1024 MCA 6 1K - 6 128 msi 2 1K - 2 64 nexusdev 4 1K - 4 16 raid_data 0 0K - 306240 16,128 linux 15 1K - 15 32,64 NFS FHA 1 1K - 1 1024 nullfs_node 0 0K - 5272989 16 nullfs_hash 1 1K - 1 64 nullfs_mount 0 0K - 103 16 tmpfs name 0 0K - 2410 16,32,64 tmpfs mount 0 0K - 1 128 journal_data 1165 139765K - 10060 512,4096 ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 128, 0, 151, 29, 153, 0, 0 UMA Zones: 224, 0, 151, 2, 153, 0, 0 UMA Slabs: 284, 0, 6687, 5, 337230, 0, 0 UMA RCntSlabs: 544, 0, 801, 116, 1117, 0, 0 UMA Hash: 128, 0, 0, 0, 2, 0, 0 16 Bucket: 76, 0, 136, 114, 252, 0, 0 32 Bucket: 140, 0, 105, 63, 268, 0, 0 64 Bucket: 268, 0, 125, 85, 404, 90, 0 128 Bucket: 524, 0, 427, 791, 59069,431777, 0 VM OBJECT: 148, 0, 16890, 98186,30714158, 0, 0 MAP: 140, 0, 8, 48, 8, 0, 0 KMAP ENTRY: 72, 57505, 7655, 1143, 9528529, 0, 0 MAP ENTRY: 72, 0, 1178, 2585,23593211, 0, 0 RADIX NODE: 68, 524320, 53014, 90586,48453425, 0, 0 fakepg: 64, 0, 0, 295, 5805, 0, 0 mt_zone: 2060, 0, 347, 57, 347, 0, 0 16: 16, 0, 4, 1620, 2133, 0, 0 16: 16, 0, 212, 194, 364, 0, 0 16: 16, 0, 3, 200, 3, 0, 0 16: 16, 0, 255, 963, 381728, 0, 0 16: 16, 0, 2762, 689, 5821475, 0, 0 16: 16, 0, 480, 1347, 2890197, 0, 0 16: 16, 0, 47, 765, 4266040, 0, 0 16: 16, 0, 49, 763, 187889, 0, 0 32: 32, 0, 33, 1097,43888283, 0, 0 32: 32, 0, 1, 2033, 2425949, 0, 0 32: 32, 0, 16, 210, 16, 0, 0 32: 32, 0, 996, 1603,15999605, 0, 0 32: 32, 0, 574, 669, 552504, 0, 0 32: 32, 0, 1460, 800, 1428030, 0, 0 32: 32, 0, 48, 22665,12885485, 0, 0 32: 32, 0, 32, 872, 1574911, 0, 0 64: 64, 0, 66, 229, 437, 0, 0 64: 64, 0, 1034, 87, 1034, 0, 0 64: 64, 0, 210, 970, 83190, 0, 0 64: 64, 0, 1766, 1774,54762187, 0, 0 64: 64, 0, 1706, 5197,42902501, 0, 0 64: 64, 0, 1775, 20409,46173828, 0, 0 64: 64, 0, 629, 3560,40688672, 0, 0 64: 64, 0, 79, 806, 406602, 0, 0 128: 128, 0, 34, 2006, 2896028, 0, 0 128: 128, 0, 15, 105, 17, 0, 0 128: 128, 0, 1, 8159, 3078635, 0, 0 128: 128, 0, 367, 413, 2992749, 0, 0 128: 128, 0, 77, 943,12393743, 0, 0 128: 128, 0, 1679, 541, 61626, 0, 0 128: 128, 0, 22, 1058, 86751, 0, 0 128: 128, 0, 97, 5003, 4612348, 0, 0 256: 256, 0, 2, 73, 15, 0, 0 256: 256, 0, 18, 57, 1549, 0, 0 256: 256, 0, 0, 0, 0, 0, 0 256: 256, 0, 739, 296, 21379, 0, 0 256: 256, 0, 86, 634,19020892, 0, 0 256: 256, 0, 1086, 4974,18309850, 0, 0 256: 256, 0, 0, 0, 0, 0, 0 256: 256, 0, 0, 75, 8, 0, 0 512: 512, 0, 5, 131, 2222079, 0, 0 512: 512, 0, 0, 0, 0, 0, 0 512: 512, 0, 0, 0, 0, 0, 0 512: 512, 0, 304, 1864, 453993, 0, 0 512: 512, 0, 312, 304, 3197953, 0, 0 512: 512, 0, 29, 19, 29, 0, 0 512: 512, 0, 3, 45, 68, 0, 0 512: 512, 0, 1, 159, 2528365, 0, 0 1024: 1024, 0, 6, 14, 7, 0, 0 1024: 1024, 0, 1, 7, 1, 0, 0 1024: 1024, 0, 0, 0, 0, 0, 0 1024: 1024, 0, 15, 81, 3438, 0, 0 1024: 1024, 0, 1, 67, 22384, 0, 0 1024: 1024, 0, 20, 168, 21148, 0, 0 1024: 1024, 0, 10, 2, 10, 0, 0 1024: 1024, 0, 4, 76, 17219, 0, 0 2048: 2048, 0, 2, 90, 3816935, 0, 0 2048: 2048, 0, 0, 0, 0, 0, 0 2048: 2048, 0, 0, 0, 0, 0, 0 2048: 2048, 0, 354, 216, 309079, 0, 0 2048: 2048, 0, 51, 37, 776, 0, 0 2048: 2048, 0, 5076, 620, 58081, 0, 0 2048: 2048, 0, 0, 0, 0, 0, 0 2048: 2048, 0, 3, 75, 51055, 0, 0 4096: 4096, 0, 10, 4, 10, 0, 0 4096: 4096, 0, 0, 0, 0, 0, 0 4096: 4096, 0, 0, 0, 0, 0, 0 4096: 4096, 0, 85, 342, 1781619, 0, 0 4096: 4096, 0, 215, 104, 61468, 0, 0 4096: 4096, 0, 256, 127, 17141, 0, 0 4096: 4096, 0, 6, 14, 15, 0, 0 4096: 4096, 0, 2, 0, 2, 0, 0 Files: 56, 0, 150, 922,17088656, 0, 0 TURNSTILE: 72, 0, 971, 169, 991, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 716, 0, 73, 507, 1676260, 0, 0 THREAD: 732, 0, 642, 328, 4000902, 0, 0 SLEEPQUEUE: 48, 0, 971, 150, 991, 0, 0 VMSPACE: 232, 0, 51, 493, 1676165, 0, 0 cpuset: 40, 0, 86, 466, 160, 0, 0 audit_record: 824, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 256, 1165, 127680, 0, 0 mbuf: 256, 0, 579, 1184,24984425, 0, 0 mbuf_cluster: 2048, 25600, 713, 805, 1630826, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 42, 2483, 0, 0 mbuf_jumbo_9k: 9216, 19200, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 12800, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 152, 0, 1203, 487,56764525, 0, 0 ttyinq: 152, 0, 330, 164, 1080, 0, 0 ttyoutq: 256, 0, 172, 113, 564, 0, 0 ata_request: 208, 0, 0, 114, 35, 0, 0 ata_composite: 180, 0, 0, 0, 0, 0, 0 VNODE: 348, 0, 16086, 73443,21425608, 0, 0 VNODEPOLL: 60, 0, 1, 188, 3, 0, 0 NAMEI: 1024, 0, 0, 768,143266287, 0, 0 S VFS Cache: 72, 0, 16273, 99532, 9165347, 0, 0 STS VFS Cache: 92, 0, 0, 15876, 595203, 0, 0 L VFS Cache: 292, 0, 11, 4695, 725432, 0, 0 LTS VFS Cache: 312, 0, 0, 372, 961, 0, 0 NCLNODE: 356, 0, 0, 2178, 346723, 0, 0 DIRHASH: 1024, 0, 0, 1248, 473665, 0, 0 Mountpoints: 732, 0, 9, 131, 17003, 0, 0 capability: 16, 0, 0, 0, 0, 0, 0 pipe: 400, 0, 13, 317, 1985575, 0, 0 ksiginfo: 80, 0, 570, 582, 21818, 0, 0 itimer: 220, 0, 1, 35, 1, 0, 0 KNOTE: 72, 0, 3, 421, 1997347, 0, 0 socket: 416, 25605, 53, 55, 1042, 0, 0 unpcb: 172, 25622, 16, 99, 232, 0, 0 ipq: 32, 904, 0, 565, 14536, 0, 0 udp_inpcb: 252, 25605, 20, 55, 709, 0, 0 udpcb: 8, 25781, 20, 589, 709, 0, 0 tcp_inpcb: 252, 25605, 16, 59, 94, 0, 0 tcpcb: 688, 25600, 16, 24, 94, 0, 0 tcptw: 52, 5184, 0, 360, 10, 0, 0 syncache: 120, 15360, 0, 160, 15, 0, 0 hostcache: 76, 15400, 1, 199, 3, 0, 0 tcpreass: 20, 1690, 0, 0, 0, 0, 0 sackhole: 20, 0, 0, 0, 0, 0, 0 sctp_ep: 932, 25600, 0, 0, 0, 0, 0 sctp_asoc: 1504, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 145, 4, 0, 0 sctp_raddr: 500, 80000, 0, 0, 0, 0, 0 sctp_chunk: 96, 400000, 0, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0, 0 sctp_stream_msg_out: 72, 400044, 0, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0 ripcb: 252, 25605, 0, 30, 1, 0, 0 rtentry: 108, 0, 17, 127, 17, 0, 0 selfd: 28, 0, 104, 912, 1091270, 0, 0 SWAPMETA: 276, 243152, 950, 11272, 146042, 0, 0 FFS inode: 116, 0, 15946, 95990,15677165, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 15946, 82499,15674065, 0, 0