GENERIC HEAD from 2015-05-02 08:31:16 UTC, r282344M, vmcore.806 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb 4718592K of memory above 4GB ignored Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1 r282344M: Sat May 2 19:03:21 CEST 2015 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 3.6.0 (tags/RELEASE_360/final 230434) 20150225 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT: running with driver "vga". CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3528859648 (3365 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... WARNING: / was not properly dismounted Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Starting file system checks: /dev/ad4s1a: 6826 files, 981924 used, 31091 free (1235 frags, 3732 blocks, 0.1% fragmentation) /dev/ad4s1f: 20984 files, 278906 used, 734109 free (9373 frags, 90592 blocks, 0.9% fragmentation) /dev/label/tmp: 388711 files, 60662847 used, 57651134 free (18398 frags, 7204092 blocks, 0.0% fragmentation) /dev/ad4s1d: 459535 files, 6236914 used, 3917285 free (267157 frags, 456266 blocks, 2.6% fragmentation) ** SU+J Recovering /dev/ad4s1e ** Reading 33554432 byte journal from inode 6. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 4 journal records in 512 bytes for 25.00% utilization ** Freed 0 inodes (0 dirs) 0 blocks, and 0 frags. ***** FILE SYSTEM MARKED CLEAN ***** Mounting local file systems:. Setting hostname: x4.osted.lan. Feeding entropy:. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=23 media: Ethernet autoselect (none) status: no carrier Starting devd. Starting pflogd: add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc46 /usr/local/lib/gcc48 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/pth /usr/local/lib/qt4 a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. Starting watchdogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Starting casperd. lock order reversal: 1st 0xe44c8e90 bufwait (bufwait) @ kern/vfs_bio.c:3134 2nd 0xc8ea3c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,c97370c0,e44c8e40,edbc8748,c0bf9dfa,...) at db_trace_self_wrapper+0x2a/frame 0xedbc8718 kdb_backtrace(c11e8511,c8ea3c00,c121bf1d,c858b8c0,c121bb62,...) at kdb_backtrace+0x2d/frame 0xedbc8780 witness_checkorder(c8ea3c00,9,c121bb59,11d,0,...) at witness_checkorder+0xd2e/frame 0xedbc87d4 _sx_xlock(c8ea3c00,0,c121bb59,11d,0,...) at _sx_xlock+0x74/frame 0xedbc8804 ufsdirhash_add(c9462708,edbc8950,fcc,edbc8890,edbc8894,...) at ufsdirhash_add+0x4a/frame 0xedbc8838 ufs_direnter(c9737000,c9ed32d0,edbc8950,edbc8b3c,e4501780,...) at ufs_direnter+0x654/frame 0xedbc88c0 ufs_mkdir(edbc8bd8,c13accdd,c1214c76,c9737000,edbc8bd4,...) at ufs_mkdir+0x7ec/frame 0xedbc8a70 VOP_MKDIR_APV(c147ccf0,edbc8bd8,edbc8b3c,edbc8b68,28803020,...) at VOP_MKDIR_APV+0x12f/frame 0xedbc8aa0 kern_mkdirat(c9e12930,ffffff9c,28803020,0,1c0) at kern_mkdirat+0x22d/frame 0xedbc8bfc sys_mkdir(c9e12930,edbc8ca8,c11d8d0d,4,c156d290,...) at sys_mkdir+0x31/frame 0xedbc8c18 syscall(edbc8ce8) at syscall+0x34a/frame 0xedbc8cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbc8cdc --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x2818f087, esp = 0xbfbfe924, ebp = 0xbfbfedd0 --- Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Starting smartd. May 2 20:00:13 x4 smartd[622]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, May 2 20:00:13 x4 smartd[622]: see the following Seagate web pages: May 2 20:00:13 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en May 2 20:00:13 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Starting default moused. Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc9737034 ufs (ufs) @ kern/vfs_subr.c:2176 2nd 0xe44c8e90 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:263 3rd 0xc9f18a0c ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,f88df630,c0ab05ef,4000,0,...) at db_trace_self_wrapper+0x2a/frame 0xf88df5d8 kdb_backtrace(c11e852a,c9f18a0c,c11ce879,c858b858,c11f1b71,...) at kdb_backtrace+0x2d/frame 0xf88df63c witness_checkorder(c9f18a0c,9,c11f1b68,880,c9f18a78,...) at witness_checkorder+0xd2e/frame 0xf88df690 __lockmgr_args(c9f18a0c,80100,c9f18a78,0,0,...) at __lockmgr_args+0x935/frame 0xf88df76c ffs_lock(f88df7e8,c8589978,c15aaa60,c8589978,c8583178,...) at ffs_lock+0x98/frame 0xf88df7a4 VOP_LOCK1_APV(c147ccf0,f88df7e8,0,c9377000,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xf88df7d0 _vn_lock(c9f189d8,80100,c11f1b68,880,c11f0d88,...) at _vn_lock+0xba/frame 0xf88df810 vget(c9f189d8,80100,c9377000,57,80100,...) at vget+0x74/frame 0xf88df848 vfs_hash_get(c9740bb0,c0fc00,80000,c9377000,f88df940,...) at vfs_hash_get+0xe7/frame 0xf88df874 ffs_vgetf(c9740bb0,c0fc00,80000,f88df940,1,...) at ffs_vgetf+0x44/frame 0xf88df8d0 softdep_sync_buf(c9737000,e44c8e40,1,0,0,...) at softdep_sync_buf+0x36b/frame 0xf88df958 ffs_syncvnode(c9737000,1,0,5fa,c8583588,...) at ffs_syncvnode+0x2aa/frame 0xf88df9b0 ffs_sync(c9740bb0,4,c11f362c,733,c9377000,...) at ffs_sync+0x1e9/frame 0xf88dfa38 vfs_write_suspend(c9740bb0,0,c121c10a,79,72,...) at vfs_write_suspend+0x1bf/frame 0xf88dfa74 vfs_write_suspend_umnt(c9740bb0,c9377000,f88dfac0,c11f1b68,8ce,...) at vfs_write_suspend_umnt+0x42/frame 0xf88dfa9c ffs_unmount(c9740bb0,8080000,f88dfb20,510,c15aaa60,...) at ffs_unmount+0xa7/frame 0xf88dfae8 dounmount(c9740bb0,8080000,c9377000,492,14f,...) at dounmount+0x4c9/frame 0xf88dfb48 sys_unmount(c9377000,f88dfca8,c11d8d0d,4,c156cb90,...) at sys_unmount+0x2d4/frame 0xf88dfc18 syscall(f88dfce8) at syscall+0x34a/frame 0xf88dfcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88dfcdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe69c, ebp = 0xbfbfe768 --- lock order reversal: 1st 0xc9457304 ufs (ufs) @ kern/vfs_mount.c:1229 2nd 0xc8d8c304 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1904 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,c15aaa60,8,10,c14559f0,...) at db_trace_self_wrapper+0x2a/frame 0xf88df740 kdb_backtrace(c11e8511,c8d8c304,c11dab9f,c858b6b8,c1217734,...) at kdb_backtrace+0x2d/frame 0xf88df7a4 witness_checkorder(c8d8c304,9,c121772b,770,0,...) at witness_checkorder+0xd2e/frame 0xf88df7f8 __lockmgr_args(c8d8c304,80400,c8d8c370,0,0,0,c121772b,770) at __lockmgr_args+0x935/frame 0xf88df8d4 vop_stdlock(f88df948,c96e4600,f88df940,c0b5bf89,c96e4600,...) at vop_stdlock+0x53/frame 0xf88df904 VOP_LOCK1_APV(c14451b8,f88df948,f88df970,c0dd4435,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xf88df930 _vn_lock(c8d8c2d0,80400,c121772b,770,f88dfa0c,...) at _vn_lock+0xba/frame 0xf88df970 softdep_flushworklist(c9740bb0,f88dfa08,c9377000,60e,c8583588,...) at softdep_flushworklist+0x68/frame 0xf88df9b0 ffs_sync(c9740bb0,4,c11f362c,733,c9377000,...) at ffs_sync+0x2ae/frame 0xf88dfa38 vfs_write_suspend(c9740bb0,0,c121c10a,79,72,...) at vfs_write_suspend+0x1bf/frame 0xf88dfa74 vfs_write_suspend_umnt(c9740bb0,c9377000,f88dfac0,c11f1b68,8ce,...) at vfs_write_suspend_umnt+0x42/frame 0xf88dfa9c ffs_unmount(c9740bb0,8080000,f88dfb20,510,c15aaa60,...) at ffs_unmount+0xa7/frame 0xf88dfae8 dounmount(c9740bb0,8080000,c9377000,492,14f,...) at dounmount+0x4c9/frame 0xf88dfb48 sys_unmount(c9377000,f88dfca8,c11d8d0d,4,c156cb90,...) at sys_unmount+0x2d4/frame 0xf88dfc18 syscall(f88dfce8) at syscall+0x34a/frame 0xf88dfcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88dfcdc --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280da46b, esp = 0xbfbfe69c, ebp = 0xbfbfe768 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 388711 files, 60662847 used, 57651134 free (18398 frags, 7204092 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp. Starting inetd. Sat May 2 20:02:21 CEST 2015 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: May 2 20:18:22 x4 su: pho to root on /dev/pts/0 20150502 20:18:49 all (1/2): quota1.sh May 2 20:19:27 x4 kernel: pid 1205 (mkdir), uid 0 inumber 12274 on /mnt: out of inodes May 2 20:19:35 x4 kernel: pid 1207 (mkdir), uid 0 inumber 12186 on /mnt: out of inodes May 2 20:20:48 x4 kernel: pid 1206 (mkdir), uid 0 inumber 16297 on /mnt: out of inodes pho = 0 0 0 20150502 20:28:59 all (2/2): fdgrowtable.sh pho = 7200 0 0 20150502 20:45:54 all (1/2): quota1.sh May 2 20:47:41 x4 kernel: pid 3279 (creat), uid 0 inumber 8196 on /mnt: out of inodes pho = 7200 0 0 20150502 20:55:58 all (2/2): fdgrowtable.sh pho = 15749 0 0 20150502 21:42:55 all (1/2): quota1.sh May 2 21:45:45 x4 kernel: pid 5209 (mkdir), uid 0 inumber 12285 on /mnt: out of inodes May 2 21:45:48 x4 kernel: pid 5192 (creat), uid 0 inumber 8264 on /mnt: out of inodes May 2 21:47:11 x4 kernel: pid 5278 (creat), uid 0 inumber 8243 on /mnt: out of inodes May 2 21:48:02 x4 kernel: pid 5284 (creat), uid 0 inumber 8246 on /mnt: out of inodes May 2 21:49:28 x4 kernel: pid 5363 (rw), uid 0 inumber 8291 on /mnt: filesystem full May 2 21:49:28 x4 kernel: pid 5362 (rw), uid 0 inumber 8215 on /mnt: filesystem full May 2 21:50:07 x4 kernel: pid 5342 (rw), uid 0 inumber 8249 on /mnt: filesystem full May 2 21:50:52 x4 kernel: pid 5348 (rw), uid 0 inumber 8216 on /mnt: filesystem full May 2 21:50:52 x4 kernel: pid 5361 (rw), uid 0 inumber 8218 on /mnt: filesystem full May 2 21:51:10 x4 kernel: pid 5424 (rw), uid 0 inumber 4129 on /mnt: filesystem full pho = 15749 0 0 20150502 21:53:10 all (2/2): fdgrowtable.sh pho = 24106 0 0 20150502 21:57:24 all (1/1): pts.sh pho = 24106 0 0 20150502 22:20:53 all (1/315): alternativeFlushPath.sh pho = 24106 0 0 20150502 22:40:59 all (2/315): callout_reset_on.sh Expensive timeout(9) function: 0xc0d36cb0(0xcdc53308) 0.013572236 s Expensive timeout(9) function: 0xc089e420(0xc8c2e000) 0.014359741 s pho = 24106 0 0 20150502 22:51:00 all (3/315): cluster.sh pho = 24106 0 0 20150502 22:51:01 all (4/315): cmp.sh pho = 42321 0 0 20150502 22:53:33 all (5/315): contigmalloc.sh pho = 42321 0 0 20150502 22:55:36 all (6/315): core.sh pho = 42321 0 0 20150502 22:58:03 all (7/315): core2.sh May 2 22:59:32 x4 kernel: Failed to write core file for process core2 (error 5) pho = 42321 0 0 20150502 22:59:32 all (8/315): core3.sh May 2 22:59:32 x4 last message repeated 25 times May 2 22:59:33 x4 kernel: Failed to write core file for process core3 (error 5) May 2 22:59:34 x4 last message repeated 207 times May 2 22:59:34 x4 mountd[537]: can't delete exports for /mnt: Device busy May 2 22:59:34 x4 kernel: Failed to write core file for process core3 (error 5) May 2 22:59:34 x4 last message repeated 109 times May 2 22:59:34 x4 mountd[537]: can't delete exports for /mnt: Device busy : : May 2 23:09:30 x4 mountd[537]: can't delete exports for /mnt: Device busy May 2 23:09:30 x4 kernel: Failed to write core file for process core3 (error 5) May 2 23:09:32 x4 last message repeated 510 times pho = 42321 0 0 20150502 23:19:32 all (9/315): core4.sh May 2 23:19:33 x4 kernel: Failed to write core file for process core4 (error 5) pho = 42321 0 0 20150502 23:19:53 all (10/315): core5.sh May 2 23:19:53 x4 last message repeated 59 times pho = 42321 0 0 20150502 23:29:55 all (11/315): crossmp.sh May 2 23:30:34 x4 mountd[537]: can't delete exports for /mnt5: Device busy May 2 23:46:22 x4 mountd[537]: can't delete exports for /mnt7: Device busy pho = 42321 0 0 20150502 23:47:06 all (12/315): crossmp2.sh May 2 23:47:09 x4 mountd[537]: bad exports list line /mnt May 2 23:47:42 x4 last message repeated 3 times May 2 23:48:10 x4 last message repeated 7 times pho = 42321 0 0 20150502 23:53:29 all (13/315): crossmp3.sh pho = 43356 0 0 20150503 00:02:37 all (14/315): crossmp4.sh pho = 49437 0 0 20150503 00:10:42 all (15/315): crossmp5.sh pho = 49437 0 0 20150503 00:12:51 all (16/315): dangling.sh pho = 49437 0 0 20150503 00:25:46 all (17/315): datamove.sh pho = 49437 0 0 20150503 00:25:49 all (18/315): datamove2.sh lock order reversal: 1st 0xcab55304 syncer (syncer) @ kern/vfs_subr.c:1769 2nd 0xd01ec5d4 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,c0ab05ef,800,0,200,...) at db_trace_self_wrapper+0x2a/frame 0xedb06928 kdb_backtrace(c11e8511,d01ec5d4,c11ce879,c858b858,c11f1b71,...) at kdb_backtrace+0x2d/frame 0xedb0698c witness_checkorder(d01ec5d4,9,c11f1b68,880,d01ec640,...) at witness_checkorder+0xd2e/frame 0xedb069e0 __lockmgr_args(d01ec5d4,80500,d01ec640,0,0,...) at __lockmgr_args+0x935/frame 0xedb06abc ffs_lock(edb06b38,246,c1fa7bd0,c8efc000,c8583588,...) at ffs_lock+0x98/frame 0xedb06af4 VOP_LOCK1_APV(c147ccf0,edb06b38,c8efc000,c15d839c,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xedb06b20 _vn_lock(d01ec5a0,80500,c11f1b68,880,c11f1b68,...) at _vn_lock+0xba/frame 0xedb06b60 vget(d01ec5a0,80500,c8efc000,edb06bf8,edb06be4,...) at vget+0x74/frame 0xedb06b98 vfs_msync(c9740000,2,2,c9740000,c145ff74,...) at vfs_msync+0xca/frame 0xedb06be4 sync_fsync(edb06c88,c13acbb8,80400,c11f1b68,6e9,...) at sync_fsync+0xe2/frame 0xedb06c08 VOP_FSYNC_APV(c145ff74,edb06c88,c11f1b68,6e9,0,...) at VOP_FSYNC_APV+0x12f/frame 0xedb06c38 sched_sync(0,edb06ce8,c11d90ec,3dc,0,...) at sched_sync+0x441/frame 0xedb06cac fork_exit(c0c1e060,0,edb06ce8) at fork_exit+0x7e/frame 0xedb06cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb06cd4 --- trap 0, eip = 0, esp = 0xedb06d20, ebp = 0 --- pho = 72502 0 0 20150503 00:46:05 all (19/315): datamove3.sh pho = 72502 0 0 20150503 00:46:08 all (20/315): datamove4.sh pho = 72502 0 0 20150503 00:46:15 all (21/315): db.sh pho = 72502 0 0 20150503 00:49:16 all (22/315): devfd.sh pho = 72502 0 0 20150503 00:49:29 all (23/315): devfs.sh May 3 00:49:34 x4 mountd[537]: can't delete exports for /mnt12: Device busy May 3 00:49:39 x4 mountd[537]: can't delete exports for /mnt11: Device busy May 3 00:49:40 x4 mountd[537]: can't delete exports for /mnt8: Device busy May 3 00:49:47 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:49:58 x4 mountd[537]: can't delete exports for /mnt9: Device busy May 3 00:50:01 x4 mountd[537]: can't delete exports for /mnt12: Device busy May 3 00:50:05 x4 mountd[537]: can't delete exports for /mnt2: Device busy May 3 00:50:10 x4 mountd[537]: can't delete exports for /mnt12: Device busy May 3 00:50:28 x4 mountd[537]: can't delete exports for /mnt13: Device busy May 3 00:50:30 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:50:32 x4 mountd[537]: can't delete exports for /mnt15: Device busy May 3 00:50:32 x4 mountd[537]: can't delete exports for /mnt6: Device busy May 3 00:50:39 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:50:53 x4 mountd[537]: can't delete exports for /mnt8: Device busy May 3 00:50:54 x4 mountd[537]: can't delete exports for /mnt14: Device busy May 3 00:50:56 x4 mountd[537]: can't delete exports for /mnt13: Device busy May 3 00:51:03 x4 mountd[537]: can't delete exports for /mnt11: Device busy May 3 00:51:10 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:51:22 x4 mountd[537]: can't delete exports for /mnt4: Device busy May 3 00:51:32 x4 mountd[537]: can't delete exports for /mnt12: Device busy May 3 00:51:36 x4 mountd[537]: can't delete exports for /mnt6: Device busy May 3 00:51:39 x4 mountd[537]: can't delete exports for /mnt8: Device busy May 3 00:51:39 x4 mountd[537]: can't delete exports for /mnt9: Device busy May 3 00:51:42 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:51:48 x4 mountd[537]: can't delete exports for /mnt2: Device busy May 3 00:51:51 x4 mountd[537]: can't delete exports for /mnt6: Device busy May 3 00:51:51 x4 mountd[537]: can't delete exports for /mnt13: Device busy May 3 00:51:55 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:51:55 x4 mountd[537]: can't delete exports for /mnt1: Device busy May 3 00:51:55 x4 mountd[537]: can't delete exports for /mnt1: Device busy May 3 00:52:07 x4 mountd[537]: can't delete exports for /mnt5: Device busy May 3 00:52:07 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:52:17 x4 mountd[537]: can't delete exports for /mnt9: Device busy May 3 00:52:35 x4 mountd[537]: can't delete exports for /mnt1: Device busy May 3 00:52:43 x4 mountd[537]: can't delete exports for /mnt5: Device busy May 3 00:52:53 x4 mountd[537]: can't delete exports for /mnt7: Device busy May 3 00:52:54 x4 mountd[537]: can't delete exports for /mnt13: Device busy May 3 00:53:02 x4 mountd[537]: can't delete exports for /mnt7: Device busy May 3 00:53:08 x4 mountd[537]: can't delete exports for /mnt10: Device busy May 3 00:53:38 x4 mountd[537]: can't delete exports for /mnt2: Device busy May 3 00:53:41 x4 mountd[537]: can't delete exports for /mnt6: Device busy May 3 00:53:45 x4 mountd[537]: can't delete exports for /mnt15: Device busy May 3 00:53:47 x4 mountd[537]: can't delete exports for /mnt12: Device busy May 3 00:53:53 x4 mountd[537]: can't delete exports for /mnt13: Device busy May 3 00:54:01 x4 mountd[537]: can't delete exports for /mnt1: Device busy May 3 00:54:03 x4 mountd[537]: can't delete exports for /mnt4: Device busy May 3 00:54:05 x4 mountd[537]: can't delete exports for /mnt14: Device busy May 3 00:54:19 x4 mountd[537]: can't delete exports for /mnt8: Device busy May 3 00:54:25 x4 mountd[537]: can't delete exports for /mnt6: Device busy May 3 00:54:28 x4 mountd[537]: can't delete exports for /mnt13: Device busy pho = 72502 0 0 20150503 00:54:29 all (24/315): devfs2.sh pho = 72502 0 0 20150503 00:54:30 all (25/315): dup2.sh pho = 72502 0 0 20150503 00:54:31 all (26/315): execi386.sh pho = 72502 0 0 20150503 00:54:32 all (27/315): exlock.sh pho = 72502 0 0 20150503 00:58:17 all (28/315): extattr.sh pho = 72502 0 0 20150503 00:58:23 all (29/315): extattr_set_fd.sh pho = 72502 0 0 20150503 00:58:24 all (30/315): extattrctl.sh lock order reversal: 1st 0xd74a071c ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xd085673c ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1681 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,a343131,c0b89800,c156cb90,4,...) at db_trace_self_wrapper+0x2a/frame 0xc8211610 kdb_backtrace(c11e8511,d085673c,c11ce879,c858b858,c121aea6,...) at kdb_backtrace+0x2d/frame 0xc8211674 witness_checkorder(d085673c,9,c121ae9d,691,0,...) at witness_checkorder+0xd2e/frame 0xc82116c8 __lockmgr_args(d085673c,80000,0,0,0,...) at __lockmgr_args+0x935/frame 0xc82117a8 ffs_vgetf(d156f2ec,2,80000,c821183c,0) at ffs_vgetf+0x171/frame 0xc8211804 ffs_vget(d156f2ec,2,80000,c821183c,0,...) at ffs_vget+0x2f/frame 0xc8211824 ufs_root(d156f2ec,80000,c8211878,72,d156f2ec,...) at ufs_root+0x4a/frame 0xc821184c ufs_extattr_autostart(d156f2ec,cf24fc40,c121ae9d,424,0,...) at ufs_extattr_autostart+0x80/frame 0xc8211888 ffs_mount(d156f2ec,c147c9f0,d7617400,c93b1a00,c88cc7c0,...) at ffs_mount+0x26de/frame 0xc82119e8 vfs_donmount(cf24fc40,0,0,d96c1b00,d96c1b00,...) at vfs_donmount+0x1344/frame 0xc8211bf0 sys_nmount(cf24fc40,c8211ca8,c11d8d0d,4,c1066357,...) at sys_nmount+0x7a/frame 0xc8211c18 syscall(c8211ce8) at syscall+0x34a/frame 0xc8211cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc8211cdc --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x280edabb, esp = 0xbfbfd8c8, ebp = 0xbfbfde20 --- pho = 72502 0 0 20150503 00:58:25 all (31/315): f_offset.sh pho = 72502 0 0 20150503 01:01:53 all (32/315): fcntl.sh pho = 72502 0 0 20150503 01:01:57 all (33/315): fdescfs.sh May 3 01:03:14 x4 mountd[537]: can't delete exports for /mnt3: Device busy May 3 01:04:57 x4 mountd[537]: can't delete exports for /mnt1: Device busy May 3 01:05:04 x4 mountd[537]: can't delete exports for /mnt4: Device busy May 3 01:05:43 x4 mountd[537]: can't delete exports for /mnt10: Device busy pho = 72502 0 0 20150503 01:06:58 all (34/315): fdgrowtable.sh pho = 86530 0 0 20150503 01:11:09 all (35/315): ffs_blkfree.sh lock order reversal: 1st 0xe44d4110 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:263 2nd 0xd7d7cd1c snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2298 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,c0c15c79,c147ccf0,f89326a0,cfa51438,...) at db_trace_self_wrapper+0x2a/frame 0xf8932680 kdb_backtrace(c11e8511,d7d7cd1c,c1217276,c858c5c0,c1216e98,...) at kdb_backtrace+0x2d/frame 0xf89326e4 witness_checkorder(d7d7cd1c,9,c1216e8f,8fa,d00cda78,...) at witness_checkorder+0xd2e/frame 0xf8932738 __lockmgr_args(d7d7cd1c,80900,d00cda78,0,0,...) at __lockmgr_args+0x935/frame 0xf8932818 ffs_copyonwrite(d00cd9d8,e44d40c0,e44d40c0,cc9dd780,41c0,...) at ffs_copyonwrite+0x1cf/frame 0xf893288c ffs_geom_strategy(d00cda98,e44d40c0,f8932938,c936e620,f89328f0,...) at ffs_geom_strategy+0xd2/frame 0xf89328a8 ufs_strategy(f8932938,c13acfef,4,0,1000,...) at ufs_strategy+0xb1/frame 0xf89328f0 VOP_STRATEGY_APV(c147ccf0,f8932938,a51508,c5531838,c14923b4,...) at VOP_STRATEGY_APV+0x13e/frame 0xf8932920 bufstrategy(cfa514f8,e44d40c0,117d,cfa514f8,f8932994,...) at bufstrategy+0x4d/frame 0xf8932950 bufwrite(e44d40c0,e44d40c0,1,0,0,...) at bufwrite+0x160/frame 0xf8932994 ffs_syncvnode(cfa51438,1,0,0,c147ccf0,...) at ffs_syncvnode+0x2f3/frame 0xf89329ec ffs_fsync(f8932a60,c13acbb8,0,cc9dd780,cfa51444,...) at ffs_fsync+0x26/frame 0xf8932a10 VOP_FSYNC_APV(c147ccf0,f8932a60,400800c,0,cfa51438,...) at VOP_FSYNC_APV+0x12f/frame 0xf8932a40 ufs_remove(f8932bd8,c13acc06,ade,f8932bd4,10,...) at ufs_remove+0xdf/frame 0xf8932a84 VOP_REMOVE_APV(c147ccf0,f8932bd8,cb3af2d0,f8932bac,bfbfe3f8,...) at VOP_REMOVE_APV+0x148/frame 0xf8932ab0 kern_unlinkat(c936e620,ffffff9c,bfbfe3f8,0,0) at kern_unlinkat+0x266/frame 0xf8932bfc sys_unlink(c936e620,f8932ca8,c11d8d0d,4,0,...) at sys_unlink+0x32/frame 0xf8932c18 syscall(f8932ce8) at syscall+0x34a/frame 0xf8932cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8932cdc --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x281974b7, esp = 0xbfbfdf8c, ebp = 0xbfbfe874 --- lock order reversal: 1st 0xd7d7cd1c snaplk (snaplk) @ kern/vfs_vnops.c:434 2nd 0xcb3af304 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1629 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,c936e620,cb3af3a0,0,40,...) at db_trace_self_wrapper+0x2a/frame 0xf8932548 kdb_backtrace(c11e8511,cb3af304,c11ce879,c858b858,c1216e98,...) at kdb_backtrace+0x2d/frame 0xf89325ac witness_checkorder(cb3af304,9,c1216e8f,65d,0,...) at witness_checkorder+0xd2e/frame 0xf8932600 __lockmgr_args(cb3af304,80000,0,0,0,...) at __lockmgr_args+0x935/frame 0xf89326e0 ffs_snapremove(cb3af2d0,c1216d60,c8587008,c15a7fe0,c15a7fc0,...) at ffs_snapremove+0x119/frame 0xf8932740 ffs_truncate(cb3af2d0,0,0,c00,0,...) at ffs_truncate+0x2a1/frame 0xf89328f0 ufs_inactive(f8932978,c13ace79,8,c11f1b68,9db,...) at ufs_inactive+0x1ed/frame 0xf8932920 VOP_INACTIVE_APV(c147ccf0,f8932978,c11f1b68,9db,d7d7cd1c,...) at VOP_INACTIVE_APV+0x12f/frame 0xf8932950 vinactive(cb3af2d0,c936e620,c11f1b68,8ce,f89329e0,...) at vinactive+0x182/frame 0xf8932998 vputx(cb3af2d0,f8932a00,c11f362c,1b2,2457918,...) at vputx+0x3c3/frame 0xf89329e0 vn_close(cb3af2d0,1,d7c03d00,c936e620,c11d8e30,...) at vn_close+0xdd/frame 0xf8932a28 vn_closefile(d19b94d0,c936e620,246,0,d19b94d0,...) at vn_closefile+0x4f/frame 0xf8932a8c _fdrop(d19b94d0,c936e620,100,cc412270,c15a7fd0,...) at _fdrop+0x21/frame 0xf8932aa8 closef(d19b94d0,c936e620,81c,80f,8,...) at closef+0x205/frame 0xf8932b10 fdescfree(c936e620,0,c11d8e30,157,f8932c18,...) at fdescfree+0x486/frame 0xf8932ba8 exit1(c936e620,0,f8932cdc,c105f11a,c936e620,...) at exit1+0x594/frame 0xf8932c08 sys_sys_exit(c936e620,f8932ca8,c11d8d0d,4,c1066357,...) at sys_sys_exit+0x1d/frame 0xf8932c18 syscall(f8932ce8) at syscall+0x34a/frame 0xf8932cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8932cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x28140aa3, esp = 0xbfbfdf28, ebp = 0xbfbfdf38 --- pho = 86530 0 0 20150503 01:11:35 all (36/315): ffs_sync.sh pho = 86530 0 0 20150503 01:14:43 all (37/315): ffs_syncvnode.sh May 3 01:19:17 x4 kernel: pid 68373 (rw), uid 1004 inumber 206278 on /mnt: filesystem full May 3 01:24:59 x4 kernel: pid 68525 (rw), uid 1004 inumber 196623 on /mnt: filesystem full May 3 01:33:06 x4 kernel: pid 68800 (rw), uid 1004 inumber 196617 on /mnt: filesystem full pho = 90476 0 0 *** Excessive run time: ffs_syncvnode.sh, 32 min 20150503 01:47:24 all (38/315): ffs_syncvnode2.sh May 3 01:47:36 x4 kernel: pid 69379 (creat), uid 1004 inumber 131106 on /mnt: filesystem full May 3 01:47:36 x4 kernel: pid 69366 (rw), uid 1004 inumber 131096 on /mnt: filesystem full May 3 01:49:46 x4 kernel: pid 69427 (rw), uid 1004 inumber 141757 on /mnt: filesystem full May 3 01:51:37 x4 kernel: pid 69464 (rw), uid 1004 inumber 131119 on /mnt: filesystem full May 3 01:51:38 x4 kernel: pid 69460 (mkdir), uid 1004 inumber 134659 on /mnt: filesystem full May 3 01:51:38 x4 kernel: pid 69475 (creat), uid 1004 inumber 131097 on /mnt: filesystem full May 3 01:51:46 x4 kernel: pid 69459 (rw), uid 1004 inumber 131117 on /mnt: filesystem full May 3 01:59:45 x4 kernel: pid 69771 (rw), uid 1004 inumber 196627 on /mnt: filesystem full May 3 01:59:53 x4 kernel: pid 69766 (mkdir), uid 1004 inumber 38 on /mnt: filesystem full May 3 01:59:53 x4 kernel: pid 69760 (rw), uid 1004 inumber 196622 on /mnt: filesystem full May 3 02:02:05 x4 kernel: pid 70002 (rw), uid 1004 inumber 64 on /mnt: filesystem full May 3 02:02:05 x4 kernel: pid 70007 (mkdir), uid 1004 inumber 769 on /mnt: filesystem full May 3 02:02:05 x4 kernel: pid 70018 (creat), uid 1004 inumber 44 on /mnt: filesystem full May 3 02:03:57 x4 kernel: pid 70677 (mkdir), uid 1004 inumber 133760 on /mnt: filesystem full May 3 02:03:57 x4 kernel: pid 70687 (rw), uid 1004 inumber 131113 on /mnt: filesystem full May 3 02:03:58 x4 kernel: pid 70678 (creat), uid 1004 inumber 131133 on /mnt: filesystem full May 3 02:10:01 x4 kernel: pid 73533 (rw), uid 1004 inumber 131109 on /mnt: filesystem full May 3 02:10:22 x4 kernel: pid 73530 (rw), uid 1004 inumber 217845 on /mnt: filesystem full May 3 02:10:33 x4 kernel: pid 73526 (rw), uid 1004 inumber 166926 on /mnt: filesystem full May 3 02:10:54 x4 kernel: pid 73524 (rw), uid 1004 inumber 173587 on /mnt: filesystem full May 3 02:11:07 x4 kernel: pid 73531 (creat), uid 1004 inumber 131101 on /mnt: filesystem full May 3 02:11:37 x4 kernel: pid 73529 (creat), uid 1004 inumber 131118 on /mnt: filesystem full May 3 02:14:14 x4 kernel: pid 75416 (rw), uid 1004 inumber 196645 on /mnt: filesystem full May 3 02:14:34 x4 kernel: pid 76380 (rw), uid 1004 inumber 196678 on /mnt: filesystem full May 3 02:14:34 x4 kernel: pid 76382 (creat), uid 1004 inumber 196654 on /mnt: filesystem full May 3 02:14:34 x4 kernel: pid 76372 (mkdir), uid 1004 inumber 755 on /mnt: filesystem full May 3 02:14:37 x4 kernel: pid 76387 (rw), uid 1004 inumber 196686 on /mnt: filesystem full pho = 93413 0 0 *** Excessive run time: ffs_syncvnode2.sh, 31 min 20150503 02:18:32 all (39/315): fifo.sh pho = 93413 0 0 20150503 02:18:34 all (40/315): fifo2.sh pho = 93413 0 0 20150503 02:20:45 all (41/315): fifo3.sh pho = 93413 0 0 20150503 02:24:01 all (42/315): flock_open_close.sh pho = 93413 0 0 20150503 02:30:39 all (43/315): fork.sh pho = 93413 0 0 20150503 02:30:45 all (44/315): fork2.sh pho = 93413 0 0 20150503 02:33:55 all (45/315): fpclone.sh pho = 93413 0 0 20150503 02:36:00 all (46/315): fpclone2.sh pho = 93413 0 0 20150503 02:36:03 all (47/315): fpu.sh pho = 93413 0 0 20150503 02:38:44 all (48/315): fragments.sh May 3 02:38:54 x4 kernel: pid 88749 (fragments), uid 1004 inumber 83871 on /mnt: filesystem full May 3 02:38:54 x4 kernel: pid 88749 (fragments), uid 1004 inumber 65554 on /mnt: filesystem full May 3 02:38:55 x4 kernel: pid 88756 (fragments), uid 1004 inumber 7156 on /mnt: filesystem full : : May 3 03:08:51 x4 kernel: pid 89237 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 3 03:08:52 x4 kernel: pid 89240 (fragments), uid 1004 inumber 2 on /mnt: filesystem full May 3 03:08:53 x4 kernel: pid 89237 (fragments), uid 1004 inumber 7300 on /mnt: filesystem full May 3 03:08:54 x4 kernel: pid 89238 (fragments), uid 1004 inumber 7190 on /mnt: filesystem full pho = 93413 0 0 20150503 03:08:56 all (49/315): fs.sh May 3 03:12:58 x4 kernel: pid 89640 (rw), uid 1004 inumber 6 on /mnt: filesystem full May 3 03:13:09 x4 kernel: pid 89648 (creat), uid 1004 inumber 2311 on /mnt: out of inodes May 3 03:13:10 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:13:11 x4 kernel: pid 89651 (creat), uid 1004 inumber 2304 on /mnt: out of inodes May 3 03:13:13 x4 kernel: pid 89652 (creat), uid 1004 inumber 2312 on /mnt: out of inodes May 3 03:13:14 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:13:17 x4 kernel: pid 89654 (creat), uid 1004 inumber 768 on /mnt: out of inodes May 3 03:13:18 x4 kernel: pid 89653 (creat), uid 1004 inumber 2310 on /mnt: out of inodes May 3 03:13:20 x4 kernel: pid 89657 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:13:26 x4 kernel: pid 89649 (creat), uid 1004 inumber 5 on /mnt: out of inodes May 3 03:13:27 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:13:28 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:13:30 x4 kernel: pid 89649 (creat), uid 1004 inumber 5 on /mnt: out of inodes May 3 03:13:31 x4 kernel: pid 89646 (creat), uid 1004 inumber 1541 on /mnt: out of inodes May 3 03:13:33 x4 kernel: pid 89657 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:13:39 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:13:40 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:13:41 x4 kernel: pid 89653 (creat), uid 1004 inumber 2310 on /mnt: out of inodes May 3 03:13:43 x4 kernel: pid 89644 (creat), uid 1004 inumber 769 on /mnt: out of inodes May 3 03:13:44 x4 kernel: pid 89653 (creat), uid 1004 inumber 2310 on /mnt: out of inodes May 3 03:13:46 x4 kernel: pid 89644 (creat), uid 1004 inumber 769 on /mnt: out of inodes May 3 03:13:47 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:13:49 x4 kernel: pid 89646 (creat), uid 1004 inumber 1541 on /mnt: out of inodes May 3 03:13:50 x4 kernel: pid 89657 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:13:51 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:13:53 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:13:54 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:13:55 x4 kernel: pid 89649 (creat), uid 1004 inumber 5 on /mnt: out of inodes May 3 03:13:57 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:13:59 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:01 x4 kernel: pid 89652 (creat), uid 1004 inumber 2312 on /mnt: out of inodes May 3 03:14:02 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:03 x4 kernel: pid 89656 (creat), uid 1004 inumber 2308 on /mnt: out of inodes May 3 03:14:05 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:14:06 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:10 x4 kernel: pid 89644 (creat), uid 1004 inumber 769 on /mnt: out of inodes May 3 03:14:15 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:14:16 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:14:18 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:14:20 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:14:23 x4 kernel: pid 89657 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:14:24 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:26 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:14:27 x4 kernel: pid 89643 (creat), uid 1004 inumber 2313 on /mnt: out of inodes May 3 03:14:28 x4 kernel: pid 89647 (creat), uid 1004 inumber 1545 on /mnt: out of inodes May 3 03:14:30 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:31 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:14:33 x4 kernel: pid 89646 (creat), uid 1004 inumber 1541 on /mnt: out of inodes May 3 03:14:34 x4 kernel: pid 89648 (creat), uid 1004 inumber 2311 on /mnt: out of inodes May 3 03:14:36 x4 kernel: pid 89646 (creat), uid 1004 inumber 1541 on /mnt: out of inodes May 3 03:14:40 x4 kernel: pid 89645 (creat), uid 1004 inumber 1547 on /mnt: out of inodes May 3 03:14:42 x4 kernel: pid 89651 (creat), uid 1004 inumber 2304 on /mnt: out of inodes May 3 03:14:44 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:14:46 x4 kernel: pid 89653 (creat), uid 1004 inumber 2310 on /mnt: out of inodes May 3 03:14:49 x4 kernel: pid 89654 (creat), uid 1004 inumber 768 on /mnt: out of inodes May 3 03:14:50 x4 kernel: pid 89654 (creat), uid 1004 inumber 768 on /mnt: out of inodes May 3 03:14:52 x4 kernel: pid 89651 (creat), uid 1004 inumber 2304 on /mnt: out of inodes May 3 03:14:53 x4 kernel: pid 89644 (creat), uid 1004 inumber 769 on /mnt: out of inodes May 3 03:14:57 x4 kernel: pid 89655 (creat), uid 1004 inumber 770 on /mnt: out of inodes May 3 03:14:58 x4 kernel: pid 89650 (creat), uid 1004 inumber 8 on /mnt: out of inodes May 3 03:21:01 x4 kernel: pid 89989 (rw), uid 1004 inumber 653 on /mnt: filesystem full May 3 03:21:09 x4 kernel: pid 89996 (rw), uid 1004 inumber 1441 on /mnt: filesystem full May 3 03:21:09 x4 kernel: pid 90001 (mkdir), uid 1004 inumber 753 on /mnt: filesystem full May 3 03:21:09 x4 kernel: pid 89997 (rw), uid 1004 inumber 1377 on /mnt: filesystem full May 3 03:22:59 x4 kernel: pid 90113 (rw), uid 1004 inumber 641 on /mnt: filesystem full May 3 03:25:08 x4 kernel: pid 90201 (creat), uid 1004 inumber 1926 on /mnt: out of inodes May 3 03:25:09 x4 kernel: pid 90212 (creat), uid 1004 inumber 642 on /mnt: out of inodes May 3 03:25:11 x4 kernel: pid 90203 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:25:15 x4 kernel: pid 90200 (creat), uid 1004 inumber 6 on /mnt: out of inodes May 3 03:25:17 x4 kernel: pid 90214 (creat), uid 1004 inumber 1923 on /mnt: out of inodes May 3 03:25:32 x4 kernel: pid 90198 (creat), uid 1004 inumber 1920 on /mnt: out of inodes May 3 03:25:35 x4 kernel: pid 90197 (creat), uid 1004 inumber 1283 on /mnt: out of inodes May 3 03:25:43 x4 kernel: pid 90204 (creat), uid 1004 inumber 1925 on /mnt: out of inodes May 3 03:25:47 x4 kernel: pid 90197 (creat), uid 1004 inumber 1283 on /mnt: out of inodes May 3 03:25:49 x4 kernel: pid 90202 (creat), uid 1004 inumber 4 on /mnt: out of inodes May 3 03:25:50 x4 kernel: pid 90198 (creat), uid 1004 inumber 1920 on /mnt: out of inodes May 3 03:25:54 x4 kernel: pid 90204 (creat), uid 1004 inumber 1925 on /mnt: out of inodes May 3 03:25:59 x4 kernel: pid 90199 (creat), uid 1004 inumber 1922 on /mnt: out of inodes May 3 03:26:01 x4 kernel: pid 90214 (creat), uid 1004 inumber 1923 on /mnt: out of inodes May 3 03:26:02 x4 kernel: pid 90214 (creat), uid 1004 inumber 1923 on /mnt: out of inodes May 3 03:26:07 x4 kernel: pid 90203 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:26:09 x4 kernel: pid 90208 (creat), uid 1004 inumber 1924 on /mnt: out of inodes May 3 03:26:13 x4 kernel: pid 90209 (creat), uid 1004 inumber 1928 on /mnt: out of inodes May 3 03:26:14 x4 kernel: pid 90203 (creat), uid 1004 inumber 7 on /mnt: out of inodes May 3 03:26:18 x4 kernel: pid 90201 (creat), uid 1004 inumber 1926 on /mnt: out of inodes May 3 03:26:29 x4 kernel: pid 90199 (creat), uid 1004 inumber 1922 on /mnt: out of inodes May 3 03:26:33 x4 kernel: pid 90207 (creat), uid 1004 inumber 640 on /mnt: out of inodes May 3 03:26:36 x4 kernel: pid 90204 (creat), uid 1004 inumber 1925 on /mnt: out of inodes May 3 03:26:40 x4 kernel: pid 90209 (creat), uid 1004 inumber 1928 on /mnt: out of inodes May 3 03:26:42 x4 kernel: pid 90200 (creat), uid 1004 inumber 6 on /mnt: out of inodes May 3 03:26:44 x4 kernel: pid 90214 (creat), uid 1004 inumber 1923 on /mnt: out of inodes May 3 03:26:48 x4 kernel: pid 90213 (creat), uid 1004 inumber 5 on /mnt: out of inodes May 3 03:26:53 x4 kernel: pid 90207 (creat), uid 1004 inumber 640 on /mnt: out of inodes May 3 03:26:55 x4 kernel: pid 90208 (creat), uid 1004 inumber 1924 on /mnt: out of inodes May 3 03:26:56 x4 kernel: pid 90192 (rw), uid 1004 inumber 1748 on /mnt: filesystem full May 3 03:26:57 x4 kernel: pid 90200 (creat), uid 1004 inumber 6 on /mnt: out of inodes May 3 03:27:00 x4 kernel: pid 90210 (creat), uid 1004 inumber 1927 on /mnt: out of inodes May 3 03:31:08 x4 kernel: pid 90606 (mkdir), uid 1004 inumber 1230 on /mnt: out of inodes May 3 03:31:14 x4 kernel: pid 90603 (creat), uid 1004 inumber 902 on /mnt: out of inodes May 3 03:31:16 x4 kernel: pid 90590 (creat), uid 1004 inumber 1618 on /mnt: out of inodes May 3 03:31:22 x4 kernel: pid 90602 (mkdir), uid 1004 inumber 964 on /mnt: out of inodes May 3 03:31:24 x4 kernel: pid 90596 (mkdir), uid 1004 inumber 2544 on /mnt: out of inodes May 3 03:31:28 x4 kernel: pid 90592 (mkdir), uid 1004 inumber 2469 on /mnt: out of inodes May 3 03:31:29 x4 kernel: pid 90604 (mkdir), uid 1004 inumber 2343 on /mnt: out of inodes May 3 03:31:32 x4 kernel: pid 90598 (mkdir), uid 1004 inumber 1912 on /mnt: out of inodes May 3 03:31:34 x4 kernel: pid 90590 (creat), uid 1004 inumber 1618 on /mnt: out of inodes May 3 03:31:38 x4 kernel: pid 90603 (creat), uid 1004 inumber 902 on /mnt: out of inodes May 3 03:31:43 x4 kernel: pid 90603 (creat), uid 1004 inumber 902 on /mnt: out of inodes May 3 03:31:45 x4 kernel: pid 90597 (creat), uid 1004 inumber 901 on /mnt: out of inodes May 3 03:31:47 x4 kernel: pid 90595 (creat), uid 1004 inumber 904 on /mnt: out of inodes May 3 03:31:52 x4 kernel: pid 90595 (creat), uid 1004 inumber 904 on /mnt: out of inodes May 3 03:32:00 x4 kernel: pid 90605 (creat), uid 1004 inumber 905 on /mnt: out of inodes May 3 03:32:04 x4 kernel: pid 90597 (creat), uid 1004 inumber 901 on /mnt: out of inodes May 3 03:32:09 x4 kernel: pid 90599 (creat), uid 1004 inumber 906 on /mnt: out of inodes May 3 03:32:11 x4 kernel: pid 90595 (creat), uid 1004 inumber 904 on /mnt: out of inodes May 3 03:32:19 x4 kernel: pid 90599 (creat), uid 1004 inumber 906 on /mnt: out of inodes May 3 03:32:24 x4 kernel: pid 90605 (creat), uid 1004 inumber 905 on /mnt: out of inodes May 3 03:32:30 x4 kernel: pid 90605 (creat), uid 1004 inumber 905 on /mnt: out of inodes May 3 03:32:38 x4 kernel: pid 90590 (creat), uid 1004 inumber 1618 on /mnt: out of inodes May 3 03:32:43 x4 kernel: pid 90590 (creat), uid 1004 inumber 1618 on /mnt: out of inodes May 3 03:32:44 x4 kernel: pid 90597 (creat), uid 1004 inumber 901 on /mnt: out of inodes May 3 03:32:55 x4 kernel: pid 90595 (creat), uid 1004 inumber 904 on /mnt: out of inodes May 3 03:32:57 x4 kernel: pid 90597 (creat), uid 1004 inumber 901 on /mnt: out of inodes May 3 03:32:58 x4 kernel: pid 90603 (creat), uid 1004 inumber 902 on /mnt: out of inodes May 3 03:33:00 x4 kernel: pid 90605 (creat), uid 1004 inumber 905 on /mnt: out of inodes May 3 03:33:01 x4 kernel: pid 90601 (creat), uid 1004 inumber 903 on /mnt: out of inodes May 3 03:35:31 x4 kernel: pid 91019 (creat), uid 1004 inumber 1298 on /mnt: out of inodes May 3 03:35:35 x4 kernel: pid 91025 (mkdir), uid 1004 inumber 391 on /mnt: out of inodes May 3 03:35:55 x4 kernel: pid 91016 (rw), uid 1004 inumber 2236 on /mnt: filesystem full May 3 03:36:06 x4 kernel: pid 91033 (mkdir), uid 1004 inumber 1278 on /mnt: out of inodes May 3 03:36:08 x4 kernel: pid 91032 (mkdir), uid 1004 inumber 436 on /mnt: out of inodes May 3 03:36:20 x4 kernel: pid 91037 (mkdir), uid 1004 inumber 1275 on /mnt: out of inodes May 3 03:37:00 x4 kernel: pid 91028 (mkdir), uid 1004 inumber 1181 on /mnt: out of inodes May 3 03:37:05 x4 kernel: pid 91222 (rw), uid 1004 inumber 1283 on /mnt: filesystem full May 3 03:37:05 x4 kernel: pid 91226 (mkdir), uid 1004 inumber 1288 on /mnt: filesystem full pho = 93413 0 0 *** Excessive run time: fs.sh, 32 min 20150503 03:41:04 all (50/315): fts.sh May 3 04:07:45 x4 kernel: pid 94943 (symlink), uid 1004 inumber 65536 on /mnt: out of inodes pho = 169181 0 0 20150503 04:11:11 all (51/315): fts2.sh pho = 169181 0 0 20150503 04:12:15 all (52/315): fullpath.sh pho = 169181 0 0 20150503 04:13:16 all (53/315): graid0.sh GEOM_STRIPE: Device data created (id=2205507153). GEOM_STRIPE: Disk md5 attached to data. GEOM_STRIPE: Disk md6 attached to data. GEOM_STRIPE: Disk md7 attached to data. GEOM_STRIPE: Device stripe/data activated. GEOM_STRIPE: Disk md7 removed from data. GEOM_STRIPE: Device stripe/data deactivated. GEOM_STRIPE: Disk md6 removed from data. GEOM_STRIPE: Disk md5 removed from data. GEOM_STRIPE: Device data destroyed. pho = 184349 0 0 20150503 04:24:24 all (54/315): graid1.sh GEOM_MIRROR: Device mirror/data launched (3/3). May 3 04:29:39 x4 kernel: pid 37073 (openat), uid 1004 inumber 32785 on /mnt: out of inodes GEOM_MIRROR: Device data: provider mirror/data destroyed. GEOM_MIRROR: Device data destroyed. pho = 211222 0 0 20150503 04:35:11 all (55/315): graid3.sh GEOM_RAID3: Device raid3/data launched (3/3). lock order reversal: 1st 0xd7db8c48 FFS (FFS Lock) @ ufs/ffs/ffs_alloc.c:1945 2nd 0xd2c0dc00 bufobj interlock (bufobj interlock) @ kern/vfs_subr.c:1675 KDB: stack backtrace: db_trace_self_wrapper(c11e4418,a3537,c15a31b8,c8583318,c8583860,...) at db_trace_self_wrapper+0x2a/frame 0xf8920448 kdb_backtrace(c11e8511,d2c0dc00,c11f1bc0,c8589a48,c11f1b71,...) at kdb_backtrace+0x2d/frame 0xf89204b0 witness_checkorder(d2c0dc00,9,c11f1b68,68b,0,...) at witness_checkorder+0xd2e/frame 0xf8920504 _rw_wlock_cookie(d2c0dc10,c11f1b68,68b,4,d2c0dc10,...) at _rw_wlock_cookie+0x78/frame 0xf8920538 brelvp(e44f51e0,c11eeb90,1e1,781,d5727ea0) at brelvp+0x3a/frame 0xf8920560 vfs_vmio_release(ffe00000,c858b788,c14559f0,3ffe0,799,...) at vfs_vmio_release+0x23e/frame 0xf8920588 brelse(e44f51e0,0,c1215e54,799,0,...) at brelse+0x58c/frame 0xf89205e0 ffs_clusteralloc(3ffe0,0,2,0,e55d4000,...) at ffs_clusteralloc+0x4d1/frame 0xf8920648 ffs_reallocblks(f89207f8,c13ad273,0,0,0,...) at ffs_reallocblks+0x1128/frame 0xf8920758 VOP_REALLOCBLKS_APV(c147ccf0,f89207f8,0,8000,0,...) at VOP_REALLOCBLKS_APV+0x135/frame 0xf8920788 cluster_write(ca12c000,e4669420,1368000,0,7f,...) at cluster_write+0x4f8/frame 0xf892082c ffs_write(f89209b0,c13aca35,ca12c034,ca12c000,c1490194,...) at ffs_write+0x412/frame 0xf89208b8 VOP_WRITE_APV(c147ccf0,f89209b0,ca12c000,373,2,...) at VOP_WRITE_APV+0x1ac/frame 0xf8920978 vn_write(d19af888,f8920be8,d7ceb200,1,c9e11930,...) at vn_write+0x328/frame 0xf89209e0 vn_io_fault_doio(c9e11930,f8920a4c,c0bad6bc,c157b164,0,...) at vn_io_fault_doio+0x34/frame 0xf8920a28 vn_io_fault1(c9e11930,0,0,ffffffff,7fffffff,...) at vn_io_fault1+0x6d/frame 0xf8920af8 vn_io_fault(d19af888,f8920be8,d7ceb200,0,c9e11930,...) at vn_io_fault+0x1fa/frame 0xf8920b4c dofilewrite(d19af888,f8920be8,ffffffff,ffffffff,0,...) at dofilewrite+0xa0/frame 0xf8920b80 kern_writev(c9e11930,4,f8920be8,c15a8908,bfbfe6c8,...) at kern_writev+0x93/frame 0xf8920bc8 sys_write(c9e11930,f8920ca8,c11d8d0d,4,c156d290,...) at sys_write+0x59/frame 0xf8920c18 syscall(f8920ce8) at syscall+0x34a/frame 0xf8920cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8920cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x281b6eef, esp = 0xbfbfd6ac, ebp = 0xbfbfe758 --- GEOM_RAID3: Device data: provider raid3/data destroyed. GEOM_RAID3: Device data destroyed. pho = 496678 1 1 20150503 04:56:58 all (56/315): holdcnt0.sh pho = 496678 1 1 20150503 04:57:01 all (57/315): inversion.sh pho = 496678 1 1 20150503 05:27:02 all (58/315): isofs.sh pho = 496678 1 1 20150503 05:27:11 all (59/315): jail.sh pho = 496678 1 1 20150503 05:27:11 all (60/315): jail2.sh pho = 496678 1 1 20150503 05:27:12 all (61/315): jail3.sh pho = 496678 1 1 20150503 05:27:13 all (62/315): jail4.sh pho = 496678 1 1 20150503 05:27:13 all (63/315): jumbo.sh Expensive timeout(9) function: 0xc0bde190(0) 0.155803457 s [zone: mbuf_jumbo_page] kern.ipc.nmbjumbop limit reached Expensive timeout(9) function: 0xc089e420(0xc8c2e000) 0.378567683 s pho = 496678 1 1 20150503 05:30:22 all (64/315): kern_umtx_inf_loop.sh pho = 496678 1 1 20150503 05:30:23 all (65/315): kevent.sh pho = 496678 1 1 20150503 05:31:33 all (66/315): kevent2.sh pho = 496678 1 1 20150503 05:31:41 all (67/315): kevent3.sh pho = 496678 1 1 20150503 05:31:42 all (68/315): kevent4.sh witness_lock_list_get: witness exhausted pho = 497354 1 1 20150503 05:33:56 all (69/315): kevent5.sh pho = 497354 1 1 20150503 05:37:56 all (70/315): kevent6.sh pho = 497354 1 1 20150503 05:38:20 all (71/315): kevent7.sh pho = 497354 1 1 20150503 05:43:21 all (72/315): kevent8.sh pho = 497354 1 1 20150503 05:45:01 all (73/315): killpg.sh pho = 497354 1 1 20150503 05:45:02 all (74/315): kinfo.sh pho = 497354 1 1 20150503 06:15:04 all (75/315): kinfo2.sh pho = 497354 1 1 20150503 06:45:06 all (76/315): kinfo3.sh pho = 497354 1 1 20150503 07:00:08 all (77/315): ldt.sh pho = 497354 1 1 20150503 07:00:09 all (78/315): ldt2.sh pho = 497354 1 1 20150503 07:00:10 all (79/315): libMicro.sh pho = 497354 1 1 20150503 07:01:10 all (80/315): linger.sh pho = 497354 1 1 20150503 07:19:50 all (81/315): linger2.sh May 3 07:19:52 x4 kernel: pid 50824 (linger2), uid 1004 inumber 891 on /mnt: filesystem full May 3 07:19:52 x4 kernel: pid 50826 (linger2), uid 1004 inumber 889 on /mnt: filesystem full pho = 497354 1 1 20150503 07:19:56 all (82/315): linger3.sh pho = 497354 1 1 20150503 07:20:06 all (83/315): linger4.sh pho = 497354 1 1 *** Excessive run time: linger4.sh, 47 min 20150503 08:07:28 all (84/315): link.sh ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start ata0: FAILURE - out of memory in start KDB: enter: Break to debugger [ thread pid 52918 tid 103374 ] Stopped at kdb_alt_break_internal+0x162: movl $0,kdb_why db> x kmem_arena kmem_arena: c157d300 db> show vmem 0xc157d300 vmem 0xc157d300 'kmem arena' quantum: 4096 size: 432013312 inuse: 432013312 free: 0 busy tags: 82501 free tags: 4 inuse size free size 4096 76015 311357440 0 0 8192 50 409600 0 0 16384 6296 103153664 0 0 32768 65 2129920 0 0 65536 34 2228224 0 0 94208 1 94208 0 0 126976 1 126976 0 0 131072 20 3125248 0 0 262144 10 2662400 0 0 524288 8 4485120 0 0 2097152 1 2240512 0 0 db> x pho1 pho1: 7bd48 db> x pho2 pho2: 1 db> x pho3 pho3: 1 db> run pho db:0:pho> bt Tracing pid 52918 tid 103374 td 0xde9dd620 kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x790400 curthread = 0xde9b0620: pid 52936 "swap" curpcb = 0xfa995d40 fpcurthread = none idlethread = 0xc8a3a000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x23296400 curthread = 0xde709000: pid 52933 "swap" curpcb = 0xfa9c1d40 fpcurthread = none idlethread = 0xc8a39c40: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x23299400 curthread = 0xde9dd620: pid 52918 "swap" curpcb = 0xfa9f4d40 fpcurthread = none idlethread = 0xc8a39930: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x2329c400 curthread = 0xde9c0620: pid 52927 "swap" curpcb = 0xfa9e2d40 fpcurthread = none idlethread = 0xc8a39620: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xc9457708: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags (VI_ACTIVE) lock type syncer: EXCL by thread 0xc8efc000 (pid 9, syncer, tid 100073) #0 0xc0b3cd6a at __lockmgr_args+0xbca #1 0xc0c06393 at vop_stdlock+0x53 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0c1e472 at sched_sync+0x412 #5 0xc0b2780e at fork_exit+0x7e #6 0xc104a3a4 at fork_trampoline+0x8 0xd19b1b40: tag ufs, type VDIR usecount 2, writecount 0, refcount 13 mountedhere 0xcf614bb0 flags (VI_ACTIVE) v_object 0xcbec84e0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0xc8efc000 (pid 9, syncer, tid 100073) with shared waiters pending #0 0xc0b3cd6a at __lockmgr_args+0xbca #1 0xc0df52d8 at ffs_lock+0x98 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0c16e64 at vget+0x74 #5 0xc0df3b60 at ffs_sync+0x510 #6 0xc0c1c2a6 at sync_fsync+0x116 #7 0xc108df2f at VOP_FSYNC_APV+0x12f #8 0xc0c1e4a1 at sched_sync+0x441 #9 0xc0b2780e at fork_exit+0x7e #10 0xc104a3a4 at fork_trampoline+0x8 ino 235521, on dev ada0s1a 0xc931fca8: tag ufs, type VDIR usecount 129, writecount 0, refcount 130 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) v_object 0xc93453a8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 11) #0 0xc0b3c931 at __lockmgr_args+0x791 #1 0xc0df52d8 at ffs_lock+0x98 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0c0b88f at lookup+0xef #5 0xc0c0b3ce at namei+0x50e #6 0xc0c2621a at kern_renameat+0x20a #7 0xc0c26009 at sys_rename+0x39 #8 0xc105f11a at syscall+0x34a #9 0xc104a391 at Xint0x80_syscall+0x21 ino 2, on dev ada0s1a 0xcf7649d8: tag ufs, type VREG usecount 3, writecount 0, refcount 5 mountedhere 0 flags (VV_TEXT|VI_ACTIVE) v_object 0xd04d2618 ref 2 pages 1 cleanbuf 0 dirtybuf 0 lock type ufs: EXCL by thread 0xdea04620 (pid 52992, sh, tid 103385) #0 0xc0b3cd6a at __lockmgr_args+0xbca #1 0xc0df52d8 at ffs_lock+0x98 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0aff997 at exec_elf32_imgact+0x717 #5 0xc0b2026b at kern_execve+0x57b #6 0xc0b1fa90 at sys_execve+0x50 #7 0xc105f11a at syscall+0x34a #8 0xc104a391 at Xint0x80_syscall+0x21 ino 70689, on dev ada0s1a 0xc9ffd870: tag ufs, type VREG usecount 8, writecount 0, refcount 10 mountedhere 0 flags (VV_TEXT|VI_ACTIVE) v_object 0xc9f7bc30 ref 4 pages 10 cleanbuf 0 dirtybuf 1 lock type ufs: SHARED (count 1) #0 0xc0b3c931 at __lockmgr_args+0x791 #1 0xc0df52d8 at ffs_lock+0x98 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0c16e64 at vget+0x74 #5 0xc0e171df at vm_fault_hold+0x83f #6 0xc0e16960 at vm_fault+0x90 #7 0xc105ea40 at trap_pfault+0x1d0 #8 0xc105e0ab at trap+0x52b #9 0xc104a2fc at calltrap+0x6 ino 1602250, on dev ada0s1d 0xd2c99b40: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags (VI_ACTIVE) v_object 0xd0c04e04 ref 0 pages 2 cleanbuf 0 dirtybuf 1 lock type ufs: EXCL by thread 0xc8efb310 (pid 473, syslogd, tid 100082) #0 0xc0b3cd6a at __lockmgr_args+0xbca #1 0xc0df52d8 at ffs_lock+0x98 #2 0xc108f42a at VOP_LOCK1_APV+0x10a #3 0xc0c29aba at _vn_lock+0xba #4 0xc0c25e92 at sys_fsync+0x152 #5 0xc105f11a at syscall+0x34a #6 0xc104a391 at Xint0x80_syscall+0x21 ino 6052868, on dev ada0s1e db:0:lockedvnods> show mount 0xc930d2ec /dev/ad4s1a on / (ufs) 0xc930d5d8 devfs on /dev (devfs) 0xc9741000 /dev/ad4s1f on /home (ufs) 0xc97408c4 /dev/ad4s1d on /usr (ufs) 0xc97405d8 /dev/ad4s1e on /var (ufs) 0xc97402ec procfs on /proc (procfs) 0xc9740000 /dev/label/tmp on /tmp (ufs) 0xcf614bb0 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 52993 46305 477 0 NE+ fork2 52992 52874 7073 0 DVL+ pfault 0xc1639a80 sh 52936 52916 52896 0 R CPU 0 swap 52935 52916 52896 0 R swap 52934 52916 52896 0 DL pfault 0xc1639a80 swap 52933 52916 52896 0 R CPU 1 swap 52932 52916 52896 0 DL pfault 0xc1639a80 swap 52931 52916 52896 0 R swap 52930 52916 52896 0 DL pfault 0xc1639a80 swap 52929 52916 52896 0 R swap 52928 52916 52896 0 R swap 52927 52916 52896 0 R CPU 3 swap 52926 52916 52896 0 R swap 52925 52916 52896 0 R swap 52924 52916 52896 0 R swap 52923 52916 52896 0 R swap 52922 52916 52896 0 R swap 52921 52916 52896 0 R swap 52920 52916 52896 0 R swap 52919 52916 52896 0 R swap 52918 52916 52896 0 R CPU 2 swap 52917 52916 52896 0 R swap 52916 52896 52896 0 SW wait 0xde69b340 swap 52915 52897 7073 0 D+ ufs 0xd19b1b74 link 52913 52897 7073 0 D+ ufs 0xd19b1b74 link 52912 52897 7073 0 D+ ufs 0xd19b1b74 link 52911 52897 7073 0 D+ ufs 0xd19b1b74 link 52910 52897 7073 0 D+ ufs 0xd19b1b74 link 52909 52897 7073 0 D+ ufs 0xd19b1b74 link 52908 52897 7073 0 D+ ufs 0xd19b1b74 link 52907 52897 7073 0 D+ ufs 0xd19b1b74 link 52906 52897 7073 0 D+ ufs 0xd19b1b74 link 52905 52897 7073 0 D+ ufs 0xd19b1b74 link 52899 52897 7073 0 D+ ufs 0xd19b1b74 link 52897 52874 7073 0 SW+ wait 0xde907340 link 52896 1 52896 0 Ss nanslp 0xc155fb41 swap 52891 0 0 0 DL mdwait 0xd1fe0000 [md5] 52875 7073 7073 0 S+ piperd 0xc9b99000 tee 52874 7073 7073 0 D+ ppwait 0xde9e8cd4 sh 7073 949 7073 0 SW+ wait 0xca01f9c0 sh 949 947 949 0 SW+ wait 0xc946a000 bash 947 946 947 0 SW+ pause 0xc8a2d3a4 csh 946 941 946 1001 SW+ wait 0xca01e340 su 941 940 941 1001 SWs+ wait 0xc946a9c0 bash 940 938 938 1001 DL biord 0xe44eed78 sshd 938 678 938 0 Ss select 0xc88e9ce4 sshd 895 1 895 0 DLs+ swread 0xcb9313fc getty 894 1 894 0 Ss+ ttyin 0xc8c77870 getty 893 1 893 0 Ss+ ttyin 0xc8c77a70 getty 892 1 892 0 Ss+ ttyin 0xc8c77c70 getty 891 1 891 0 Ss+ ttyin 0xc8c78870 getty 890 1 890 0 Ss+ ttyin 0xc8c78670 getty 889 1 889 0 Ss+ ttyin 0xc8c78470 getty 888 1 888 0 Ss+ ttyin 0xc8c78270 getty 887 1 887 0 Ss+ ttyin 0xc8c78070 getty 868 1 868 0 Ss select 0xc9475224 inetd 689 1 689 0 ?Ws cron 684 1 684 25 SWs pause 0xca01f064 sendmail 681 1 681 0 Ss select 0xc93d5f64 sendmail 678 1 678 0 Ss select 0xc93d5ee4 sshd 631 1 631 0 Ss select 0xc93d6164 moused 624 1 623 0 SW nanslp 0xc155fb43 smartd 605 1 605 0 Ss select 0xc936d2e4 powerd 602 1 602 0 Ds (threaded) ntpd 100093 D kmem are 0xc157d380 ntpd 549 543 543 0 S (threaded) nfsd 100094 S rpcsvc 0xc93b4190 nfsd: master 100105 S rpcsvc 0xc9310b90 nfsd: service 100106 S rpcsvc 0xc89f5e90 nfsd: service 100107 S rpcsvc 0xc9310a90 nfsd: service 100108 S rpcsvc 0xc89f5d90 nfsd: service 100109 S rpcsvc 0xc89f5c90 nfsd: service 100110 S rpcsvc 0xc9310990 nfsd: service 100111 S rpcsvc 0xc9310890 nfsd: service 100112 S rpcsvc 0xc89f5b90 nfsd: service 100113 S rpcsvc 0xc9310790 nfsd: service 100114 S rpcsvc 0xc89f5a90 nfsd: service 100115 S rpcsvc 0xc9310690 nfsd: service 100116 S rpcsvc 0xc89f5990 nfsd: service 100117 S rpcsvc 0xc9310590 nfsd: service 100118 S rpcsvc 0xc89f5890 nfsd: service 100119 S rpcsvc 0xc9310490 nfsd: service 100120 S rpcsvc 0xc89f5790 nfsd: service 100121 S rpcsvc 0xc9310390 nfsd: service 100122 S rpcsvc 0xc89f5690 nfsd: service 100123 S rpcsvc 0xc89f5590 nfsd: service 100124 S rpcsvc 0xc89f5490 nfsd: service 100125 S rpcsvc 0xc9310290 nfsd: service 100126 S rpcsvc 0xc9310190 nfsd: service 100127 S rpcsvc 0xc89f5390 nfsd: service 100128 S rpcsvc 0xc9310090 nfsd: service 100129 S rpcsvc 0xc89f5290 nfsd: service 100130 S rpcsvc 0xc930fe90 nfsd: service 100131 S rpcsvc 0xc930fd90 nfsd: service 100132 S rpcsvc 0xc89f5190 nfsd: service 100133 S rpcsvc 0xc930fc90 nfsd: service 100134 S rpcsvc 0xc930fb90 nfsd: service 100135 S rpcsvc 0xc89f5090 nfsd: service 543 1 543 0 Ss select 0xc9475324 nfsd 537 1 537 0 Ss select 0xc93d6124 mountd 530 526 526 0 S select 0xc936d364 nfsuserd 529 526 526 0 S select 0xc94755e4 nfsuserd 528 526 526 0 S select 0xc936d3a4 nfsuserd 527 526 526 0 S select 0xc94756a4 nfsuserd 526 1 526 0 SWs pause 0xc93686e4 nfsuserd 509 1 509 0 Ss select 0xc88e9e64 casperd 508 1 508 0 Ss select 0xc93d64a4 casperd 494 1 494 0 Ss select 0xc93d64e4 rpcbind 477 1 477 0 Ds kmem are 0xc157d380 watchdogd 473 1 473 0 Ds bo_wwait 0xd2c99c4c syslogd 349 1 349 0 Ss select 0xc9475764 rtsold 313 1 313 0 Ss select 0xc94756e4 devd 16 0 0 0 DL vlruwt 0xc91749c0 [vnlru] 9 0 0 0 DL getblk 0xe45a3da0 [syncer] 8 0 0 0 DL (threaded) [bufdaemon] 100072 D psleep 0xc15d7b04 [bufdaemon] 100084 D sdflush 0xc96e4884 [/home worker] 100086 D sdflush 0xc96e4484 [/usr worker] 100087 D sdflush 0xc96e4284 [/var worker] 100169 D sdflush 0xc92ca084 [/tmp worker] 101124 D sdflush 0xd2282484 [/mnt worker] 7 0 0 0 RL [pagezero] 6 0 0 0 DL psleep 0xc15dd3c4 [vmdaemon] 5 0 0 0 DL wswbuf0 0xc15dcc24 [pagedaemon] 4 0 0 0 DL waiting_ 0xc163789c [sctp_iterator] 3 0 0 0 DL - 0xc8c2ca3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100032 D - 0xc8c3eacc [usbus0] 100033 D - 0xc8c3eafc [usbus0] 100034 D - 0xc8c3eb2c [usbus0] 100035 D - 0xc8c3eb5c [usbus0] 100037 D - 0xc8c8bacc [usbus1] 100038 D - 0xc8c8bafc [usbus1] 100039 D - 0xc8c8bb2c [usbus1] 100040 D - 0xc8c8bb5c [usbus1] 100042 D - 0xc8c94acc [usbus2] 100043 D - 0xc8c94afc [usbus2] 100044 D - 0xc8c94b2c [usbus2] 100045 D - 0xc8c94b5c [usbus2] 100046 D - 0xc8c9aacc [usbus3] 100047 D - 0xc8c9aafc [usbus3] 100048 D - 0xc8c9ab2c [usbus3] 100049 D - 0xc8c9ab5c [usbus3] 100050 D - 0xc8ca2acc [usbus4] 100051 D - 0xc8ca2afc [usbus4] 100052 D - 0xc8ca2b2c [usbus4] 100053 D - 0xc8ca2b5c [usbus4] 100055 D - 0xc8c44ca4 [usbus5] 100056 D - 0xc8c44cd4 [usbus5] 100057 D - 0xc8c44d04 [usbus5] 100058 D - 0xc8c44d34 [usbus5] 2 0 0 0 DL (threaded) [cam] 100024 D - 0xc1495c00 [doneq0] 100068 D - 0xc1495aac [scanner] 14 0 0 0 DL - 0xc14b07d0 [rand_harvestq] 13 0 0 0 DL (threaded) [geom] 100013 D - 0xc1632a20 [g_event] 100014 D - 0xc1632a24 [g_up] 100015 D - 0xc1632a28 [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi4: clock (0)] 100008 I [swi4: clock (1)] 100009 I [swi4: clock (2)] 100010 I [swi4: clock (3)] 100011 I [swi1: netisr 0] 100012 I [swi3: vm] 100019 I [swi5: fast taskq] 100025 I [swi6: task queue] 100026 I [swi6: Giant taskq] 100028 I [irq256: hdac0] 100029 I [irq257: re0] 100030 I [irq22: ahci0] 100031 I [irq16: hdac1 ohci0] 100036 I [irq17: ohci1 ohci3] 100041 I [irq18: ohci2 ohci4] 100054 I [irq19: ehci0] 100059 I [irq14: ata0] 100060 I [swi0: uart] 100062 I [irq1: atkbd0] 100063 I [irq12: psm0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xc8a2c9c0 [init] 10 0 0 0 DL audit_wo 0xc16397d0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D vmwait 0xc1639a80 [swapper] 100016 D - 0xc857be80 [firmware taskq] 100018 D - 0xc857bb80 [thread taskq] 100020 D - 0xc857ba80 [kqueue taskq] 100021 D - 0xc857ba00 [acpi_task_0] 100022 D - 0xc857ba00 [acpi_task_1] 100023 D - 0xc857ba00 [acpi_task_2] 100027 D - 0xc857b800 [ffs_trim taskq] 100064 D - 0xc8cf3780 [mca taskq] 100065 D - 0xc155ea4d [deadlkres] 100067 D kmem are 0xc157d380 [CAM taskq] db:0:ps> allt Tracing command watchdogd pid 52993 tid 103136 td 0xde835000 sched_switch(c8583318,c15a4970,c8583318,c8583860,fd66aa40,...) at sched_switch+0x4e2/frame 0xfd66a9d4 end(0,0,0,0,0,...) at 0xc8583860/frame 0xc1fb2010 Tracing command sh pid 52992 tid 103385 td 0xdea04620 sched_switch(dea04620,0,104,1a9,faa15480,...) at sched_switch+0x4e2/frame 0xfaa15448 mi_switch(104,0,c11e6431,200,c163ae80,...) at mi_switch+0x1bf/frame 0xfaa15480 sleepq_switch(dea04620,0,c11e6431,276,dea04620,...) at sleepq_switch+0x1c5/frame 0xfaa154b4 sleepq_wait(c1639a80,78,c12205f3,0,0,...) at sleepq_wait+0x48/frame 0xfaa154d8 _sleep(c1639a80,c163ae80,278,c12205f3,0,0,0,0,100) at _sleep+0x3a1/frame 0xfaa15534 vm_waitpfault(d5f72d68,0,0,faa15684,faa15674,...) at vm_waitpfault+0x98/frame 0xfaa15560 vm_fault_hold(de6822d0,8049000,2,0,0,...) at vm_fault_hold+0x1804/frame 0xfaa156b8 vm_fault(de6822d0,8049000,2,0,2000000,...) at vm_fault+0x90/frame 0xfaa156e0 trap_pfault(8049000,d4,c14559f0,dea04620,c1fb2310,...) at trap_pfault+0x1d0/frame 0xfaa15758 trap(faa158a0) at trap+0x6be/frame 0xfaa15894 calltrap() at calltrap+0x6/frame 0xfaa15894 --- trap 0xc, eip = 0xc105c166, esp = 0xfaa158e0, ebp = 0xfaa15928 --- copyout(80499d0,134,128,3,1000,...) at copyout+0x36/frame 0xfaa15928 exec_elf32_imgact(faa15aa8,faa15b98,4200844,1,edc00000,...) at exec_elf32_imgact+0x8a4/frame 0xfaa159b0 kern_execve(dea04620,faa15be0,0,28878a7c,28878a88,...) at kern_execve+0x57b/frame 0xfaa15bc0 sys_execve(dea04620,faa15ca8,c11d8d0d,4,df,...) at sys_execve+0x50/frame 0xfaa15c18 syscall(faa15ce8) at syscall+0x34a/frame 0xfaa15cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfaa15cdc --- syscall (59, FreeBSD ELF32, sys_execve), eip = 0x2816041f, esp = 0xbfbfe494, ebp = 0xbfbfe5c4 --- Tracing command swap pid 52936 tid 103344 td 0xde9b0620 cpustop_handler(0,13,fa995cdc,c105dbb7,0,...) at cpustop_handler+0x25/frame 0xfa995b90 ipi_nmi_handler(0,0,40bee8,0,fa995be4,...) at ipi_nmi_handler+0x36/frame 0xfa995ba0 trap(fa995ce8) at trap+0x37/frame 0xfa995cdc calltrap() at calltrap+0x6/frame 0xfa995cdc --- trap 0x13, eip = 0x80495aa, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52935 tid 103262 td 0xde8f3930 sched_switch(de8f3930,0,207,1a4,de9053b2,...) at sched_switch+0x4e2/frame 0xfa852c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa852cac ast(fa852ce8) at ast+0x2ec/frame 0xfa852cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52934 tid 101840 td 0xdddac930 sched_switch(dddac930,0,104,1a9,fc81d8c8,...) at sched_switch+0x4e2/frame 0xfc81d890 mi_switch(104,0,c11e6431,200,c163ae80,...) at mi_switch+0x1bf/frame 0xfc81d8c8 sleepq_switch(dddac930,0,c11e6431,276,dddac930,...) at sleepq_switch+0x1c5/frame 0xfc81d8fc sleepq_wait(c1639a80,78,c12205f3,0,0,...) at sleepq_wait+0x48/frame 0xfc81d920 _sleep(c1639a80,c163ae80,278,c12205f3,0,0,0,0,100) at _sleep+0x3a1/frame 0xfc81d97c vm_waitpfault(d0c46e04,5279,0,fc81dacc,fc81dabc,...) at vm_waitpfault+0x98/frame 0xfc81d9a8 vm_fault_hold(d2e53c30,2de79000,2,0,0,...) at vm_fault_hold+0x1804/frame 0xfc81db00 vm_fault(d2e53c30,2de79000,2,0,2e19e19,...) at vm_fault+0x90/frame 0xfc81db28 trap_pfault(2de79000,fc81dce8,40ea05,0,fc81dbe4,...) at trap_pfault+0x1d0/frame 0xfc81dba0 trap(fc81dce8) at trap+0x52b/frame 0xfc81dcdc calltrap() at calltrap+0x6/frame 0xfc81dcdc --- trap 0xc, eip = 0x281b75a3, esp = 0xbfbfe6cc, ebp = 0xbfbfe708 --- Tracing command swap pid 52933 tid 103357 td 0xde709000 cpustop_handler(1,13,fa9c1cdc,c105dbb7,1,...) at cpustop_handler+0x25/frame 0xfa9c1b90 ipi_nmi_handler(1,0,40243e,0,fa9c1be4,...) at ipi_nmi_handler+0x36/frame 0xfa9c1ba0 trap(fa9c1ce8) at trap+0x37/frame 0xfa9c1cdc calltrap() at calltrap+0x6/frame 0xfa9c1cdc --- trap 0x13, eip = 0x80495aa, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52932 tid 100079 td 0xc8efb620 sched_switch(c8efb620,0,104,1a9,edb1f8c8,...) at sched_switch+0x4e2/frame 0xedb1f890 mi_switch(104,0,c11e6431,200,c163ae80,...) at mi_switch+0x1bf/frame 0xedb1f8c8 sleepq_switch(c8efb620,0,c11e6431,276,c8efb620,...) at sleepq_switch+0x1c5/frame 0xedb1f8fc sleepq_wait(c1639a80,78,c12205f3,0,0,...) at sleepq_wait+0x48/frame 0xedb1f920 _sleep(c1639a80,c163ae80,278,c12205f3,0,0,0,0,100) at _sleep+0x3a1/frame 0xedb1f97c vm_waitpfault(d2434444,998,0,edb1facc,edb1fabc,...) at vm_waitpfault+0x98/frame 0xedb1f9a8 vm_fault_hold(deda6000,29598000,2,0,0,...) at vm_fault_hold+0x1804/frame 0xedb1fb00 vm_fault(deda6000,29598000,2,0,207ee70,...) at vm_fault+0x90/frame 0xedb1fb28 trap_pfault(29598000,edb1fce8,40b63e,0,edb1fbe4,...) at trap_pfault+0x1d0/frame 0xedb1fba0 trap(edb1fce8) at trap+0x52b/frame 0xedb1fcdc calltrap() at calltrap+0x6/frame 0xedb1fcdc --- trap 0xc, eip = 0x80495ac, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52931 tid 103284 td 0xde935930 sched_switch(de935930,0,207,1a4,de9083b2,...) at sched_switch+0x4e2/frame 0xfa8aac74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa8aacac ast(fa8aace8) at ast+0x2ec/frame 0xfa8aacdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52930 tid 103371 td 0xde9b0c40 sched_switch(de9b0c40,0,104,1a9,fa9eb8c8,...) at sched_switch+0x4e2/frame 0xfa9eb890 mi_switch(104,0,c11e6431,200,c163ae80,...) at mi_switch+0x1bf/frame 0xfa9eb8c8 sleepq_switch(de9b0c40,0,c11e6431,276,de9b0c40,...) at sleepq_switch+0x1c5/frame 0xfa9eb8fc sleepq_wait(c1639a80,78,c12205f3,0,0,...) at sleepq_wait+0x48/frame 0xfa9eb920 _sleep(c1639a80,c163ae80,278,c12205f3,0,0,0,0,100) at _sleep+0x3a1/frame 0xfa9eb97c vm_waitpfault(d08a0618,333a,0,fa9ebacc,fa9ebabc,...) at vm_waitpfault+0x98/frame 0xfa9eb9a8 vm_fault_hold(d6265d20,2bf3a000,2,0,0,...) at vm_fault_hold+0x1804/frame 0xfa9ebb00 vm_fault(d6265d20,2bf3a000,2,0,2230cc5,...) at vm_fault+0x90/frame 0xfa9ebb28 trap_pfault(2bf3a000,fa9ebce8,40b4bb,0,fa9ebbe4,...) at trap_pfault+0x1d0/frame 0xfa9ebba0 trap(fa9ebce8) at trap+0x52b/frame 0xfa9ebcdc calltrap() at calltrap+0x6/frame 0xfa9ebcdc --- trap 0xc, eip = 0x281b75a3, esp = 0xbfbfe6cc, ebp = 0xbfbfe708 --- Tracing command swap pid 52929 tid 103356 td 0xde709310 sched_switch(de709310,0,207,1a4,de7006b2,...) at sched_switch+0x4e2/frame 0xfa9bec74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa9becac ast(fa9bece8) at ast+0x2ec/frame 0xfa9becdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52928 tid 103230 td 0xde8c8930 sched_switch(de8c8930,0,207,1a4,de8bb3b2,...) at sched_switch+0x4e2/frame 0xfd7d2c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfd7d2cac ast(fd7d2ce8) at ast+0x2ec/frame 0xfd7d2cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52927 tid 103368 td 0xde9c0620 cpustop_handler(3,13,fa9e2cdc,c105dbb7,3,...) at cpustop_handler+0x25/frame 0xfa9e2b90 ipi_nmi_handler(3,0,40d657,0,fa9e2be4,...) at ipi_nmi_handler+0x36/frame 0xfa9e2ba0 trap(fa9e2ce8) at trap+0x37/frame 0xfa9e2cdc calltrap() at calltrap+0x6/frame 0xfa9e2cdc --- trap 0x13, eip = 0x80495aa, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52926 tid 103288 td 0xde91dc40 sched_switch(de91dc40,0,207,1a4,de93f9b3,...) at sched_switch+0x4e2/frame 0xfa8bac74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa8bacac ast(fa8bace8) at ast+0x2ec/frame 0xfa8bacdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52925 tid 103219 td 0xde8b3930 sched_switch(de8b3930,0,207,1a4,de8c20b3,...) at sched_switch+0x4e2/frame 0xfd7a6c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfd7a6cac ast(fd7a6ce8) at ast+0x2ec/frame 0xfd7a6cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52924 tid 103218 td 0xde8b3c40 sched_switch(de8b3c40,0,207,1a4,de8c23b6,...) at sched_switch+0x4e2/frame 0xfd7a2c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfd7a2cac ast(fd7a2ce8) at ast+0x2ec/frame 0xfd7a2cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52923 tid 103363 td 0xde9c1620 sched_switch(de9c1620,0,207,1a4,de9cc9b2,...) at sched_switch+0x4e2/frame 0xfa9d3c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa9d3cac ast(fa9d3ce8) at ast+0x2ec/frame 0xfa9d3cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52922 tid 103354 td 0xde709930 sched_switch(de709930,0,207,1a4,de7010b1,...) at sched_switch+0x4e2/frame 0xfa9b8c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa9b8cac ast(fa9b8ce8) at ast+0x2ec/frame 0xfa9b8cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52921 tid 103358 td 0xde6f5c40 sched_switch(de6f5c40,0,207,1a4,de7000b3,...) at sched_switch+0x4e2/frame 0xfa9c4c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfa9c4cac ast(fa9c4ce8) at ast+0x2ec/frame 0xfa9c4cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52920 tid 103393 td 0xde9f6c40 sched_switch(de9f6c40,0,207,1a4,de9cd9b2,...) at sched_switch+0x4e2/frame 0xfaa2dc74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfaa2dcac ast(faa2dce8) at ast+0x2ec/frame 0xfaa2dcdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52919 tid 102972 td 0xde6e5930 sched_switch(de6e5930,0,207,1a4,de7039b2,...) at sched_switch+0x4e2/frame 0xfd463c74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfd463cac ast(fd463ce8) at ast+0x2ec/frame 0xfd463cdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52918 tid 103374 td 0xde9dd620 kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52917 tid 101839 td 0xdddacc40 sched_switch(dddacc40,0,207,1a4,ddda39b6,...) at sched_switch+0x4e2/frame 0xfc81ac74 mi_switch(207,0,c11e705f,f7,c1628188,...) at mi_switch+0x1bf/frame 0xfc81acac ast(fc81ace8) at ast+0x2ec/frame 0xfc81acdc doreti_ast() at doreti_ast+0x1a/frame 0xbfbfe7a8 Tracing command swap pid 52916 tid 102910 td 0xde6a4310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(de6a4310,ffffffff,ceb4,191fe,de6a4310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command link pid 52915 tid 103255 td 0xde90a000 sched_switch(de90a000,0,104,1a9,fa836640,...) at sched_switch+0x4e2/frame 0xfa836608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfa836640 sleepq_switch(de90a000,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfa836674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfa836698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfa8366e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfa8367bc ffs_lock(fa836838,fa83680c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfa8367f4 VOP_LOCK1_APV(c147ccf0,fa836838,0,fa8368ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfa836820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfa836860 vget(d19b1b40,202100,de90a000,298,c1490194,...) at vget+0x74/frame 0xfa836898 cache_lookup(c931fca8,fa836b00,fa836b14,0,0,...) at cache_lookup+0x5bb/frame 0xfa8368f8 vfs_cache_lookup(fa8369b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfa836940 VOP_LOOKUP_APV(c147ccf0,fa8369b0,fa836b14,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfa836970 lookup(fa836ab8,c11f0f6b,10e,d3,c108bb1e,...) at lookup+0x551/frame 0xfa8369d8 namei(fa836ab8,3,800102c,0,bfbfe463,...) at namei+0x50e/frame 0xfa836a78 kern_renameat(de90a000,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0x20a/frame 0xfa836bf8 sys_rename(de90a000,fa836ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfa836c18 syscall(fa836ce8) at syscall+0x34a/frame 0xfa836cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa836cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52913 tid 103352 td 0xde70a000 sched_switch(de70a000,0,104,1a9,fa9b2640,...) at sched_switch+0x4e2/frame 0xfa9b2608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfa9b2640 sleepq_switch(de70a000,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfa9b2674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfa9b2698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfa9b26e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfa9b27bc ffs_lock(fa9b2838,fa9b280c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfa9b27f4 VOP_LOCK1_APV(c147ccf0,fa9b2838,0,fa9b28ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfa9b2820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfa9b2860 vget(d19b1b40,202100,de70a000,298,c1490194,...) at vget+0x74/frame 0xfa9b2898 cache_lookup(c931fca8,fa9b2b00,fa9b2b14,0,0,...) at cache_lookup+0x5bb/frame 0xfa9b28f8 vfs_cache_lookup(fa9b29b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfa9b2940 VOP_LOOKUP_APV(c147ccf0,fa9b29b0,fa9b2b14,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfa9b2970 lookup(fa9b2ab8,c11f0f6b,10e,d3,c108bb1e,...) at lookup+0x551/frame 0xfa9b29d8 namei(fa9b2ab8,3,800102c,0,bfbfe062,...) at namei+0x50e/frame 0xfa9b2a78 kern_renameat(de70a000,ffffff9c,bfbfe463,ffffff9c,bfbfe062,0) at kern_renameat+0x20a/frame 0xfa9b2bf8 sys_rename(de70a000,fa9b2ca8,c11d8d0d,4,c1066357,...) at sys_rename+0x39/frame 0xfa9b2c18 syscall(fa9b2ce8) at syscall+0x34a/frame 0xfa9b2cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa9b2cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52912 tid 103292 td 0xde947310 sched_switch(de947310,0,104,1a9,fa8ca640,...) at sched_switch+0x4e2/frame 0xfa8ca608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfa8ca640 sleepq_switch(de947310,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfa8ca674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfa8ca698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfa8ca6e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfa8ca7bc ffs_lock(fa8ca838,fa8ca80c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfa8ca7f4 VOP_LOCK1_APV(c147ccf0,fa8ca838,0,fa8ca8ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfa8ca820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfa8ca860 vget(d19b1b40,202100,de947310,298,c1490194,...) at vget+0x74/frame 0xfa8ca898 cache_lookup(c931fca8,fa8cab88,fa8cab9c,0,0,...) at cache_lookup+0x5bb/frame 0xfa8ca8f8 vfs_cache_lookup(fa8ca9b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfa8ca940 VOP_LOOKUP_APV(c147ccf0,fa8ca9b0,fa8cab9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfa8ca970 lookup(fa8cab40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfa8ca9d8 namei(fa8cab40,2,400100c,0,bfbfe062,...) at namei+0x50e/frame 0xfa8caa78 kern_renameat(de947310,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0xb8/frame 0xfa8cabf8 sys_rename(de947310,fa8caca8,c11d8d0d,4,c156c490,...) at sys_rename+0x39/frame 0xfa8cac18 syscall(fa8cace8) at syscall+0x34a/frame 0xfa8cacdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa8cacdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52911 tid 101846 td 0xdddc2930 sched_switch(dddc2930,0,104,1a9,fc82f640,...) at sched_switch+0x4e2/frame 0xfc82f608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfc82f640 sleepq_switch(dddc2930,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfc82f674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfc82f698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfc82f6e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfc82f7bc ffs_lock(fc82f838,fc82f80c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfc82f7f4 VOP_LOCK1_APV(c147ccf0,fc82f838,0,fc82f8ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfc82f820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfc82f860 vget(d19b1b40,202100,dddc2930,298,c1490194,...) at vget+0x74/frame 0xfc82f898 cache_lookup(c931fca8,fc82fb88,fc82fb9c,0,0,...) at cache_lookup+0x5bb/frame 0xfc82f8f8 vfs_cache_lookup(fc82f9b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfc82f940 VOP_LOOKUP_APV(c147ccf0,fc82f9b0,fc82fb9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfc82f970 lookup(fc82fb40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfc82f9d8 namei(fc82fb40,2,400100c,0,bfbfe463,...) at namei+0x50e/frame 0xfc82fa78 kern_renameat(dddc2930,ffffff9c,bfbfe463,ffffff9c,bfbfe062,0) at kern_renameat+0xb8/frame 0xfc82fbf8 sys_rename(dddc2930,fc82fca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfc82fc18 syscall(fc82fce8) at syscall+0x34a/frame 0xfc82fcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfc82fcdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52910 tid 103225 td 0xde8c9930 sched_switch(de8c9930,0,104,1a9,fd7be640,...) at sched_switch+0x4e2/frame 0xfd7be608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfd7be640 sleepq_switch(de8c9930,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfd7be674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfd7be698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfd7be6e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfd7be7bc ffs_lock(fd7be838,fd7be80c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfd7be7f4 VOP_LOCK1_APV(c147ccf0,fd7be838,0,fd7be8ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfd7be820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfd7be860 vget(d19b1b40,202100,de8c9930,298,c1490194,...) at vget+0x74/frame 0xfd7be898 cache_lookup(c931fca8,fd7beb88,fd7beb9c,0,0,...) at cache_lookup+0x5bb/frame 0xfd7be8f8 vfs_cache_lookup(fd7be9b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfd7be940 VOP_LOOKUP_APV(c147ccf0,fd7be9b0,fd7beb9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfd7be970 lookup(fd7beb40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfd7be9d8 namei(fd7beb40,2,400100c,0,bfbfe062,...) at namei+0x50e/frame 0xfd7bea78 kern_renameat(de8c9930,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0xb8/frame 0xfd7bebf8 sys_rename(de8c9930,fd7beca8,c11d8d0d,4,c156c490,...) at sys_rename+0x39/frame 0xfd7bec18 syscall(fd7bece8) at syscall+0x34a/frame 0xfd7becdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfd7becdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52909 tid 102977 td 0xde70ac40 sched_switch(de70ac40,0,104,1a9,fd477640,...) at sched_switch+0x4e2/frame 0xfd477608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfd477640 sleepq_switch(de70ac40,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfd477674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfd477698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfd4776e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfd4777bc ffs_lock(fd477838,fd47780c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfd4777f4 VOP_LOCK1_APV(c147ccf0,fd477838,0,fd4778ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfd477820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfd477860 vget(d19b1b40,202100,de70ac40,298,c1490194,...) at vget+0x74/frame 0xfd477898 cache_lookup(c931fca8,fd477b88,fd477b9c,0,0,...) at cache_lookup+0x5bb/frame 0xfd4778f8 vfs_cache_lookup(fd4779b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfd477940 VOP_LOOKUP_APV(c147ccf0,fd4779b0,fd477b9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfd477970 lookup(fd477b40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfd4779d8 namei(fd477b40,2,400100c,0,bfbfe463,...) at namei+0x50e/frame 0xfd477a78 kern_renameat(de70ac40,ffffff9c,bfbfe463,ffffff9c,bfbfe062,0) at kern_renameat+0xb8/frame 0xfd477bf8 sys_rename(de70ac40,fd477ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfd477c18 syscall(fd477ce8) at syscall+0x34a/frame 0xfd477cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfd477cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52908 tid 103237 td 0xde8b5310 sched_switch(de8b5310,0,104,1a9,fd7ee640,...) at sched_switch+0x4e2/frame 0xfd7ee608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfd7ee640 sleepq_switch(de8b5310,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfd7ee674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfd7ee698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfd7ee6e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfd7ee7bc ffs_lock(fd7ee838,fd7ee80c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfd7ee7f4 VOP_LOCK1_APV(c147ccf0,fd7ee838,0,fd7ee8ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfd7ee820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfd7ee860 vget(d19b1b40,202100,de8b5310,298,c1490194,...) at vget+0x74/frame 0xfd7ee898 cache_lookup(c931fca8,fd7eeb00,fd7eeb14,0,0,...) at cache_lookup+0x5bb/frame 0xfd7ee8f8 vfs_cache_lookup(fd7ee9b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfd7ee940 VOP_LOOKUP_APV(c147ccf0,fd7ee9b0,fd7eeb14,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfd7ee970 lookup(fd7eeab8,c11f0f6b,10e,d3,c108bb1e,...) at lookup+0x551/frame 0xfd7ee9d8 namei(fd7eeab8,3,800102c,0,bfbfe463,...) at namei+0x50e/frame 0xfd7eea78 kern_renameat(de8b5310,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0x20a/frame 0xfd7eebf8 sys_rename(de8b5310,fd7eeca8,c11d8d0d,4,c156c490,...) at sys_rename+0x39/frame 0xfd7eec18 syscall(fd7eece8) at syscall+0x34a/frame 0xfd7eecdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfd7eecdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52907 tid 103373 td 0xde9dd930 sched_switch(de9dd930,0,104,1a9,fa9f1640,...) at sched_switch+0x4e2/frame 0xfa9f1608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfa9f1640 sleepq_switch(de9dd930,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfa9f1674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfa9f1698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfa9f16e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfa9f17bc ffs_lock(fa9f1838,fa9f180c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfa9f17f4 VOP_LOCK1_APV(c147ccf0,fa9f1838,0,fa9f18ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfa9f1820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfa9f1860 vget(d19b1b40,202100,de9dd930,298,c1490194,...) at vget+0x74/frame 0xfa9f1898 cache_lookup(c931fca8,fa9f1b88,fa9f1b9c,0,0,...) at cache_lookup+0x5bb/frame 0xfa9f18f8 vfs_cache_lookup(fa9f19b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfa9f1940 VOP_LOOKUP_APV(c147ccf0,fa9f19b0,fa9f1b9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfa9f1970 lookup(fa9f1b40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfa9f19d8 namei(fa9f1b40,2,400100c,0,bfbfe463,...) at namei+0x50e/frame 0xfa9f1a78 kern_renameat(de9dd930,ffffff9c,bfbfe463,ffffff9c,bfbfe062,0) at kern_renameat+0xb8/frame 0xfa9f1bf8 sys_rename(de9dd930,fa9f1ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfa9f1c18 syscall(fa9f1ce8) at syscall+0x34a/frame 0xfa9f1cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa9f1cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52906 tid 103353 td 0xde709c40 sched_switch(de709c40,0,104,1a9,fa9b5640,...) at sched_switch+0x4e2/frame 0xfa9b5608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfa9b5640 sleepq_switch(de709c40,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfa9b5674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfa9b5698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfa9b56e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfa9b57bc ffs_lock(fa9b5838,fa9b580c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfa9b57f4 VOP_LOCK1_APV(c147ccf0,fa9b5838,0,fa9b58ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfa9b5820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfa9b5860 vget(d19b1b40,202100,de709c40,298,c1490194,...) at vget+0x74/frame 0xfa9b5898 cache_lookup(c931fca8,fa9b5b88,fa9b5b9c,0,0,...) at cache_lookup+0x5bb/frame 0xfa9b58f8 vfs_cache_lookup(fa9b59b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfa9b5940 VOP_LOOKUP_APV(c147ccf0,fa9b59b0,fa9b5b9c,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfa9b5970 lookup(fa9b5b40,c11f0f6b,10e,d3,8ce,...) at lookup+0x551/frame 0xfa9b59d8 namei(fa9b5b40,2,400100c,0,bfbfe463,...) at namei+0x50e/frame 0xfa9b5a78 kern_renameat(de709c40,ffffff9c,bfbfe463,ffffff9c,bfbfe062,0) at kern_renameat+0xb8/frame 0xfa9b5bf8 sys_rename(de709c40,fa9b5ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfa9b5c18 syscall(fa9b5ce8) at syscall+0x34a/frame 0xfa9b5cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa9b5cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52905 tid 101434 td 0xdb84dc40 sched_switch(db84dc40,0,104,1a9,c65a3640,...) at sched_switch+0x4e2/frame 0xc65a3608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xc65a3640 sleepq_switch(db84dc40,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xc65a3674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xc65a3698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xc65a36e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xc65a37bc ffs_lock(c65a3838,c65a380c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xc65a37f4 VOP_LOCK1_APV(c147ccf0,c65a3838,0,c65a38ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xc65a3820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xc65a3860 vget(d19b1b40,202100,db84dc40,298,c1490194,...) at vget+0x74/frame 0xc65a3898 cache_lookup(c931fca8,c65a3b00,c65a3b14,0,0,...) at cache_lookup+0x5bb/frame 0xc65a38f8 vfs_cache_lookup(c65a39b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xc65a3940 VOP_LOOKUP_APV(c147ccf0,c65a39b0,c65a3b14,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xc65a3970 lookup(c65a3ab8,c11f0f6b,10e,d3,c108bb1e,...) at lookup+0x551/frame 0xc65a39d8 namei(c65a3ab8,3,800102c,0,bfbfe463,...) at namei+0x50e/frame 0xc65a3a78 kern_renameat(db84dc40,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0x20a/frame 0xc65a3bf8 sys_rename(db84dc40,c65a3ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xc65a3c18 syscall(c65a3ce8) at syscall+0x34a/frame 0xc65a3cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc65a3cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52899 tid 102929 td 0xde6bd930 sched_switch(de6bd930,0,104,1a9,fd3d2640,...) at sched_switch+0x4e2/frame 0xfd3d2608 mi_switch(104,0,c11e6431,200,d19b1be0,...) at mi_switch+0x1bf/frame 0xfd3d2640 sleepq_switch(de6bd930,0,c11e6431,276,d19b1b74,...) at sleepq_switch+0x1c5/frame 0xfd3d2674 sleepq_wait(d19b1b74,60,c11ce879,4,1,...) at sleepq_wait+0x48/frame 0xfd3d2698 sleeplk(d19b1be0,c11ce879,60,33,1,...) at sleeplk+0x1a6/frame 0xfd3d26e0 __lockmgr_args(d19b1b74,202100,d19b1be0,0,0,...) at __lockmgr_args+0x72f/frame 0xfd3d27bc ffs_lock(fd3d2838,fd3d280c,c0c06198,c931fcdc,c15d7e20,...) at ffs_lock+0x98/frame 0xfd3d27f4 VOP_LOCK1_APV(c147ccf0,fd3d2838,0,fd3d28ac,c149209c,...) at VOP_LOCK1_APV+0x10a/frame 0xfd3d2820 _vn_lock(d19b1b40,202100,c11f1b68,880,c11f024f,...) at _vn_lock+0xba/frame 0xfd3d2860 vget(d19b1b40,202100,de6bd930,298,c1490194,...) at vget+0x74/frame 0xfd3d2898 cache_lookup(c931fca8,fd3d2b00,fd3d2b14,0,0,...) at cache_lookup+0x5bb/frame 0xfd3d28f8 vfs_cache_lookup(fd3d29b0,c13ac615,c1214d3d,c11f0f6b,c1490194,...) at vfs_cache_lookup+0xb5/frame 0xfd3d2940 VOP_LOOKUP_APV(c147ccf0,fd3d29b0,fd3d2b14,1fd,c931fd48,...) at VOP_LOOKUP_APV+0x12f/frame 0xfd3d2970 lookup(fd3d2ab8,c11f0f6b,10e,d3,c108bb1e,...) at lookup+0x551/frame 0xfd3d29d8 namei(fd3d2ab8,3,800102c,0,bfbfe463,...) at namei+0x50e/frame 0xfd3d2a78 kern_renameat(de6bd930,ffffff9c,bfbfe062,ffffff9c,bfbfe463,0) at kern_renameat+0x20a/frame 0xfd3d2bf8 sys_rename(de6bd930,fd3d2ca8,c11d8d0d,4,c156cb90,...) at sys_rename+0x39/frame 0xfd3d2c18 syscall(fd3d2ce8) at syscall+0x34a/frame 0xfd3d2cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfd3d2cdc --- syscall (128, FreeBSD ELF32, sys_rename), eip = 0x280d828b, esp = 0xbfbfe048, ebp = 0xbfbfe86c --- Tracing command link pid 52897 tid 103254 td 0xde90a310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(de90a310,ffffffff,cea1,19356,de90a310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command swap pid 52896 tid 103370 td 0xde9c0000 sched_switch(de9c0000,0,104,1a9,fa9e8a84,...) at sched_switch+0x4e2/frame 0xfa9e8a4c mi_switch(104,0,c11e6431,200,ad58,...) at mi_switch+0x1bf/frame 0xfa9e8a84 sleepq_switch(de9c0000,0,c11e6431,1b6,8,...) at sleepq_switch+0x1c5/frame 0xfa9e8ab8 sleepq_catch_signals(c0badc30,de9c0000,1,201,1,...) at sleepq_catch_signals+0x382/frame 0xfa9e8af4 sleepq_timedwait_sig(c155fb41,6c,ad58,10000000,0,...) at sleepq_timedwait_sig+0x14/frame 0xfa9e8b1c _sleep(c155fb41,0,16c,c11e13c6,247be1dc,...) at _sleep+0x326/frame 0xfa9e8b78 kern_nanosleep(bfbfe780,fa9e8bf0,fa9e8bf8,fa9e8ce8,1,...) at kern_nanosleep+0x14b/frame 0xfa9e8bd8 sys_nanosleep(de9c0000,fa9e8ca8,c11d8d0d,4,c1066357,...) at sys_nanosleep+0x68/frame 0xfa9e8c18 syscall(fa9e8ce8) at syscall+0x34a/frame 0xfa9e8cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa9e8cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2814a84f, esp = 0xbfbfe774, ebp = 0xbfbfe7a8 --- Tracing command md5 pid 52891 tid 103224 td 0xde89e930 sched_switch(de89e930,0,104,1a9,fd7babb4,...) at sched_switch+0x4e2/frame 0xfd7bab7c mi_switch(104,0,c11e6431,200,d1fe0020,...) at mi_switch+0x1bf/frame 0xfd7babb4 sleepq_switch(de89e930,0,c11e6431,276,de89e930,...) at sleepq_switch+0x1c5/frame 0xfd7babe8 sleepq_wait(d1fe0000,5c,c11680aa,0,0,...) at sleepq_wait+0x48/frame 0xfd7bac0c _sleep(d1fe0000,d1fe0020,25c,c11680aa,0,...) at _sleep+0x3a1/frame 0xfd7bac68 md_kthread(d1fe0000,fd7bace8,c11d90ec,3dc,0,...) at md_kthread+0x1c7/frame 0xfd7bacac fork_exit(c07a8f40,d1fe0000,fd7bace8) at fork_exit+0x7e/frame 0xfd7bacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xfd7bacd4 --- trap 0, eip = 0, esp = 0xfd7bad20, ebp = 0 --- Tracing command tee pid 52875 tid 103348 td 0xde9af930 sched_switch(de9af930,0,104,1a9,fa9a5a18,...) at sched_switch+0x4e2/frame 0xfa9a59e0 mi_switch(104,0,c11e6431,200,d19a3d10,...) at mi_switch+0x1bf/frame 0xfa9a5a18 sleepq_switch(de9af930,0,c11e6431,1b6,c11e6431,...) at sleepq_switch+0x1c5/frame 0xfa9a5a4c sleepq_catch_signals(c11e6431,16d,c9b99190,d80,de9af930,...) at sleepq_catch_signals+0x382/frame 0xfa9a5a88 sleepq_wait_sig(c9b99000,5c,c11e9b90,100,0,...) at sleepq_wait_sig+0x14/frame 0xfa9a5aac _sleep(c9b99000,c9b99180,15c,c11e9b90,0,...) at _sleep+0x356/frame 0xfa9a5b08 pipe_read(cd6034d0,fa9a5be8,c93b1a00,0,de9af930,...) at pipe_read+0x2da/frame 0xfa9a5b4c dofileread(cd6034d0,fa9a5be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xfa9a5b80 kern_readv(de9af930,0,fa9a5be8,c0e258f9,28803000,...) at kern_readv+0x93/frame 0xfa9a5bc8 sys_read(de9af930,fa9a5ca8,c11d8d0d,4,c1066357,...) at sys_read+0x59/frame 0xfa9a5c18 syscall(fa9a5ce8) at syscall+0x34a/frame 0xfa9a5cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfa9a5cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281a1f0f, esp = 0xbfbfe858, ebp = 0xbfbfe884 --- Tracing command sh pid 52874 tid 103228 td 0xde8c9000 sched_switch(de8c9000,0,104,1a9,fd7cab70,...) at sched_switch+0x4e2/frame 0xfd7cab38 mi_switch(104,0,c11e6431,200,de9e8a68,...) at mi_switch+0x1bf/frame 0xfd7cab70 sleepq_switch(de8c9000,0,c11e6431,299,0,...) at sleepq_switch+0x1c5/frame 0xfd7caba4 sleepq_timedwait(de9e8cd4,0,fd7cac04,0,0,...) at sleepq_timedwait+0x48/frame 0xfd7cabc8 _cv_timedwait_sbt(de9e8cd4,de9e8a68,fffffed8,0,0,...) at _cv_timedwait_sbt+0x23c/frame 0xfd7cac18 syscall(fd7cace8) at syscall+0x86f/frame 0xfd7cacdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xfd7cacdc --- syscall (52992), eip = 0x2815fd70, esp = 0xbfbfe5f8, ebp = 0xbfbfe650 --- Tracing command sh pid 7073 tid 100196 td 0xcbdbf310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(cbdbf310,ffffffff,1ba1,18764,cbdbf310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command bash pid 949 tid 100164 td 0xca245930 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(ca245930,ffffffff,3b5,18744,ca245930,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command csh pid 947 tid 100152 td 0xc9e11c40 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c9e11c40,ffffffff,3b3,18738,c9e11c40,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command su pid 946 tid 100163 td 0xc9f77930 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c9f77930,ffffffff,3b2,18743,c9f77930,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command bash pid 941 tid 100157 td 0xca246310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(ca246310,ffffffff,3ad,1873d,ca246310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command sshd pid 940 tid 100155 td 0xc9e11310 sched_switch(c9e11310,0,104,1a9,f8926710,...) at sched_switch+0x4e2/frame 0xf89266d8 mi_switch(104,0,c11e6431,200,c88b73d0,...) at mi_switch+0x1bf/frame 0xf8926710 sleepq_switch(c9e11310,0,c11e6431,276,c9e11310,...) at sleepq_switch+0x1c5/frame 0xf8926744 sleepq_wait(e44eed78,5c,c11eefff,0,0,...) at sleepq_wait+0x48/frame 0xf8926768 _sleep(e44eed78,c88b73d0,5c,c11eefff,0,...) at _sleep+0x3a1/frame 0xf89267c4 bwait(e44eed78,5c,c11eefff,f892688c,f89268c0,...) at bwait+0x8f/frame 0xf8926800 bufwait(e44eed78,e44eed78,ffffffff,4000,0,...) at bufwait+0x38/frame 0xf8926818 ufs_bmaparray(c9ffd870,d,0,f89268e8,0,...) at ufs_bmaparray+0x41f/frame 0xf89268c0 ufs_bmap(f8926970,c13acf9c,8,c12229ae,149,...) at ufs_bmap+0x68/frame 0xf8926908 VOP_BMAP_APV(c147ccf0,f8926970,149,0,8,...) at VOP_BMAP_APV+0x135/frame 0xf8926938 vnode_pager_haspage(c9f7bc30,34,0,f8926ae0,f8926ae8,...) at vnode_pager_haspage+0x166/frame 0xf89269a8 vm_fault_hold(c9360960,807c000,1,0,0,...) at vm_fault_hold+0x919/frame 0xf8926b00 vm_fault(c9360960,807c000,1,0,1000000,...) at vm_fault+0x90/frame 0xf8926b28 trap_pfault(807c001,4000000,c94264d0,f8926ca8,0,...) at trap_pfault+0x1d0/frame 0xf8926ba0 trap(f8926ce8) at trap+0x52b/frame 0xf8926cdc calltrap() at calltrap+0x6/frame 0xf8926cdc --- trap 0xc, eip = 0x807c001, esp = 0xbfbfa804, ebp = 0xbfbfa81c --- Tracing command sshd pid 938 tid 100091 td 0xc9476930 sched_switch(c9476930,0,104,1a9,edb76950,...) at sched_switch+0x4e2/frame 0xedb76918 mi_switch(104,0,c11e6431,200,edb769bc,...) at mi_switch+0x1bf/frame 0xedb76950 sleepq_switch(c9476930,0,c11e6431,1b6,edb769c8,...) at sleepq_switch+0x1c5/frame 0xedb76984 sleepq_catch_signals(edb769e4,c0b454f9,c88e9ce0,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedb769c0 sleepq_wait_sig(c88e9ce4,0,edb76a1c,101,0,...) at sleepq_wait_sig+0x14/frame 0xedb769e4 _cv_wait_sig(c88e9ce4,c88e9cd0,c11e991b,732,62d,...) at _cv_wait_sig+0x214/frame 0xedb76a30 seltdwait(ffffffff,0,0,c9476930,0,...) at seltdwait+0xbb/frame 0xedb76a60 kern_poll(c9476930,bfbfe860,1,0,0,...) at kern_poll+0x4f0/frame 0xedb76be0 sys_poll(c9476930,edb76ca8,c11d8d0d,4,c1066357,...) at sys_poll+0x71/frame 0xedb76c18 syscall(edb76ce8) at syscall+0x34a/frame 0xedb76cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb76cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x28487c53, esp = 0xbfbfe82c, ebp = 0xbfbfe848 --- Tracing command getty pid 895 tid 100180 td 0xcbad8310 sched_switch(cbad8310,0,104,1a9,f8986520,...) at sched_switch+0x4e2/frame 0xf89864e8 mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xf8986520 sleepq_switch(cbad8310,0,c11e6431,299,cbad8310,...) at sleepq_switch+0x1c5/frame 0xf8986554 sleepq_timedwait(cb9313fc,50,13,0,0,...) at sleepq_timedwait+0x48/frame 0xf8986578 _sleep(cb9313fc,cb9313a8,50,c121d57d,ffffe8e0,...) at _sleep+0x33d/frame 0xf89865d4 swap_pager_getpages(cb9313a8,f8986694,3,2,f8986768,...) at swap_pager_getpages+0x465/frame 0xf8986628 vm_fault_hold(c93e6870,bfbfe000,2,0,0,...) at vm_fault_hold+0xd02/frame 0xf8986780 vm_fault(c93e6870,bfbfe000,2,0,2b6d79b,...) at vm_fault+0x90/frame 0xf89867a8 trap_pfault(bfbfedf7,c0b86d8d,c157c04c,f8986858,7123c945,...) at trap_pfault+0x1d0/frame 0xf8986820 trap(f8986968) at trap+0x6be/frame 0xf898695c calltrap() at calltrap+0x6/frame 0xf898695c --- trap 0xc, eip = 0xc105c16d, esp = 0xf89869a8, ebp = 0xf89869dc --- copyout(f8986be8,0) at copyout+0x3d/frame 0xf89869dc uiomove(cbad1ac8,1,f8986be8,e0,cbaae340,...) at uiomove+0x1f/frame 0xf89869ec ttyinq_read_uio(c8c77e28,c8c77e00,f8986be8,1,0,...) at ttyinq_read_uio+0x13b/frame 0xf8986aa8 ttydisc_read(c8c77e00,f8986be8,0,b2,f8986be8,...) at ttydisc_read+0x37e/frame 0xf8986af0 ttydev_read(c8cbca00,f8986be8,0,0,f8986b40,...) at ttydev_read+0x55/frame 0xf8986b10 devfs_read_f(c9426a80,f8986be8,c93b4000,0,cbad8310,...) at devfs_read_f+0xa8/frame 0xf8986b4c dofileread(c9426a80,f8986be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8986b80 kern_readv(cbad8310,0,f8986be8,c15aabc0,bfbfedf7,...) at kern_readv+0x93/frame 0xf8986bc8 sys_read(cbad8310,f8986ca8,c11d8d0d,4,c1066357,...) at sys_read+0x59/frame 0xf8986c18 syscall(f8986ce8) at syscall+0x34a/frame 0xf8986cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8986cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 894 tid 100179 td 0xcbad8620 sched_switch(cbad8620,0,104,1a9,f89839a4,...) at sched_switch+0x4e2/frame 0xf898396c mi_switch(104,0,c11e6431,200,c15aaab8,...) at mi_switch+0x1bf/frame 0xf89839a4 sleepq_switch(cbad8620,0,c11e6431,1b6,f8983a1c,...) at sleepq_switch+0x1c5/frame 0xf89839d8 sleepq_catch_signals(f8983a38,c0b454f9,c8c77814,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8983a14 sleepq_wait_sig(c8c77870,0,f8983a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8983a38 _cv_wait_sig(c8c77870,c8c77804,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf8983a84 tty_wait(c8c77800,c8c77870,f8983be8,1,0,...) at tty_wait+0x4b/frame 0xf8983aa8 ttydisc_read(c8c77800,f8983be8,0,b2,f8983be8,...) at ttydisc_read+0x3d4/frame 0xf8983af0 ttydev_read(c9025c00,f8983be8,0,0,f8983b40,...) at ttydev_read+0x55/frame 0xf8983b10 devfs_read_f(c9386070,f8983be8,c93b1300,0,cbad8620,...) at devfs_read_f+0xa8/frame 0xf8983b4c dofileread(c9386070,f8983be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8983b80 kern_readv(cbad8620,0,f8983be8,c15aaab8,bfbfedf7,...) at kern_readv+0x93/frame 0xf8983bc8 sys_read(cbad8620,f8983ca8,c11d8d0d,4,0,...) at sys_read+0x59/frame 0xf8983c18 syscall(f8983ce8) at syscall+0x34a/frame 0xf8983cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8983cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 893 tid 100178 td 0xc9f79c40 sched_switch(c9f79c40,0,104,1a9,f89809a4,...) at sched_switch+0x4e2/frame 0xf898096c mi_switch(104,0,c11e6431,200,c15aaa60,...) at mi_switch+0x1bf/frame 0xf89809a4 sleepq_switch(c9f79c40,0,c11e6431,1b6,f8980a1c,...) at sleepq_switch+0x1c5/frame 0xf89809d8 sleepq_catch_signals(f8980a38,c0b454f9,c8c77a14,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8980a14 sleepq_wait_sig(c8c77a70,0,f8980a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8980a38 _cv_wait_sig(c8c77a70,c8c77a04,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf8980a84 tty_wait(c8c77a00,c8c77a70,f8980be8,1,0,...) at tty_wait+0x4b/frame 0xf8980aa8 ttydisc_read(c8c77a00,f8980be8,0,b2,f8980be8,...) at ttydisc_read+0x3d4/frame 0xf8980af0 ttydev_read(c9025e00,f8980be8,0,0,f8980b40,...) at ttydev_read+0x55/frame 0xf8980b10 devfs_read_f(c9426968,f8980be8,c89f6d00,0,c9f79c40,...) at devfs_read_f+0xa8/frame 0xf8980b4c dofileread(c9426968,f8980be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8980b80 kern_readv(c9f79c40,0,f8980be8,c15aaa60,bfbfedf7,...) at kern_readv+0x93/frame 0xf8980bc8 sys_read(c9f79c40,f8980ca8,c11d8d0d,4,0,...) at sys_read+0x59/frame 0xf8980c18 syscall(f8980ce8) at syscall+0x34a/frame 0xf8980cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8980cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 892 tid 100177 td 0xca244000 sched_switch(ca244000,0,104,1a9,f897d9a4,...) at sched_switch+0x4e2/frame 0xf897d96c mi_switch(104,0,c11e6431,200,c15abfb0,...) at mi_switch+0x1bf/frame 0xf897d9a4 sleepq_switch(ca244000,0,c11e6431,1b6,f897da1c,...) at sleepq_switch+0x1c5/frame 0xf897d9d8 sleepq_catch_signals(f897da38,c0b454f9,c8c77c14,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf897da14 sleepq_wait_sig(c8c77c70,0,f897da70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf897da38 _cv_wait_sig(c8c77c70,c8c77c04,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf897da84 tty_wait(c8c77c00,c8c77c70,f897dbe8,1,0,...) at tty_wait+0x4b/frame 0xf897daa8 ttydisc_read(c8c77c00,f897dbe8,0,b2,f897dbe8,...) at ttydisc_read+0x3d4/frame 0xf897daf0 ttydev_read(c9026100,f897dbe8,0,0,f897db40,...) at ttydev_read+0x55/frame 0xf897db10 devfs_read_f(c9426bd0,f897dbe8,c89f7300,0,ca244000,...) at devfs_read_f+0xa8/frame 0xf897db4c dofileread(c9426bd0,f897dbe8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf897db80 kern_readv(ca244000,0,f897dbe8,c15abfb0,bfbfedf7,...) at kern_readv+0x93/frame 0xf897dbc8 sys_read(ca244000,f897dca8,c11d8d0d,4,0,...) at sys_read+0x59/frame 0xf897dc18 syscall(f897dce8) at syscall+0x34a/frame 0xf897dcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf897dcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 891 tid 100176 td 0xca244310 sched_switch(ca244310,0,104,1a9,f897a9a4,...) at sched_switch+0x4e2/frame 0xf897a96c mi_switch(104,0,c11e6431,200,c15aa9b0,...) at mi_switch+0x1bf/frame 0xf897a9a4 sleepq_switch(ca244310,0,c11e6431,1b6,f897aa1c,...) at sleepq_switch+0x1c5/frame 0xf897a9d8 sleepq_catch_signals(f897aa38,c0b454f9,c8c78814,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf897aa14 sleepq_wait_sig(c8c78870,0,f897aa70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf897aa38 _cv_wait_sig(c8c78870,c8c78804,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf897aa84 tty_wait(c8c78800,c8c78870,f897abe8,1,0,...) at tty_wait+0x4b/frame 0xf897aaa8 ttydisc_read(c8c78800,f897abe8,0,b2,f897abe8,...) at ttydisc_read+0x3d4/frame 0xf897aaf0 ttydev_read(c9026300,f897abe8,0,0,f897ab40,...) at ttydev_read+0x55/frame 0xf897ab10 devfs_read_f(c93f0af0,f897abe8,c93b1c00,0,ca244310,...) at devfs_read_f+0xa8/frame 0xf897ab4c dofileread(c93f0af0,f897abe8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf897ab80 kern_readv(ca244310,0,f897abe8,c15aa9b0,bfbfedf7,...) at kern_readv+0x93/frame 0xf897abc8 sys_read(ca244310,f897aca8,c11d8d0d,4,c156c490,...) at sys_read+0x59/frame 0xf897ac18 syscall(f897ace8) at syscall+0x34a/frame 0xf897acdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf897acdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 890 tid 100175 td 0xca244620 sched_switch(ca244620,0,104,1a9,f89779a4,...) at sched_switch+0x4e2/frame 0xf897796c mi_switch(104,0,c11e6431,200,c15aa6f0,...) at mi_switch+0x1bf/frame 0xf89779a4 sleepq_switch(ca244620,0,c11e6431,1b6,f8977a1c,...) at sleepq_switch+0x1c5/frame 0xf89779d8 sleepq_catch_signals(f8977a38,c0b454f9,c8c78614,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8977a14 sleepq_wait_sig(c8c78670,0,f8977a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8977a38 _cv_wait_sig(c8c78670,c8c78604,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf8977a84 tty_wait(c8c78600,c8c78670,f8977be8,1,0,...) at tty_wait+0x4b/frame 0xf8977aa8 ttydisc_read(c8c78600,f8977be8,0,b2,f8977be8,...) at ttydisc_read+0x3d4/frame 0xf8977af0 ttydev_read(c9026500,f8977be8,0,0,f8977b40,...) at ttydev_read+0x55/frame 0xf8977b10 devfs_read_f(c93f07a8,f8977be8,c930f100,0,ca244620,...) at devfs_read_f+0xa8/frame 0xf8977b4c dofileread(c93f07a8,f8977be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8977b80 kern_readv(ca244620,0,f8977be8,c15aa6f0,bfbfedf7,...) at kern_readv+0x93/frame 0xf8977bc8 sys_read(ca244620,f8977ca8,c11d8d0d,4,c1066357,...) at sys_read+0x59/frame 0xf8977c18 syscall(f8977ce8) at syscall+0x34a/frame 0xf8977cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8977cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 889 tid 100174 td 0xca244930 sched_switch(ca244930,0,104,1a9,f89749a4,...) at sched_switch+0x4e2/frame 0xf897496c mi_switch(104,0,c11e6431,200,c15aa640,...) at mi_switch+0x1bf/frame 0xf89749a4 sleepq_switch(ca244930,0,c11e6431,1b6,f8974a1c,...) at sleepq_switch+0x1c5/frame 0xf89749d8 sleepq_catch_signals(f8974a38,c0b454f9,c8c78414,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8974a14 sleepq_wait_sig(c8c78470,0,f8974a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8974a38 _cv_wait_sig(c8c78470,c8c78404,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf8974a84 tty_wait(c8c78400,c8c78470,f8974be8,1,0,...) at tty_wait+0x4b/frame 0xf8974aa8 ttydisc_read(c8c78400,f8974be8,0,b2,f8974be8,...) at ttydisc_read+0x3d4/frame 0xf8974af0 ttydev_read(c8cc0500,f8974be8,0,0,f8974b40,...) at ttydev_read+0x55/frame 0xf8974b10 devfs_read_f(c93f0bd0,f8974be8,c93b3600,0,ca244930,...) at devfs_read_f+0xa8/frame 0xf8974b4c dofileread(c93f0bd0,f8974be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8974b80 kern_readv(ca244930,0,f8974be8,c15aa640,bfbfedf7,...) at kern_readv+0x93/frame 0xf8974bc8 sys_read(ca244930,f8974ca8,c11d8d0d,4,c11e2492,...) at sys_read+0x59/frame 0xf8974c18 syscall(f8974ce8) at syscall+0x34a/frame 0xf8974cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8974cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 888 tid 100173 td 0xca244c40 sched_switch(ca244c40,0,104,1a9,f89719a4,...) at sched_switch+0x4e2/frame 0xf897196c mi_switch(104,0,c11e6431,200,c15abae0,...) at mi_switch+0x1bf/frame 0xf89719a4 sleepq_switch(ca244c40,0,c11e6431,1b6,f8971a1c,...) at sleepq_switch+0x1c5/frame 0xf89719d8 sleepq_catch_signals(f8971a38,c0b454f9,c8c78214,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8971a14 sleepq_wait_sig(c8c78270,0,f8971a70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8971a38 _cv_wait_sig(c8c78270,c8c78204,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf8971a84 tty_wait(c8c78200,c8c78270,f8971be8,1,0,...) at tty_wait+0x4b/frame 0xf8971aa8 ttydisc_read(c8c78200,f8971be8,0,b2,f8971be8,...) at ttydisc_read+0x3d4/frame 0xf8971af0 ttydev_read(c8cc0700,f8971be8,0,0,f8971b40,...) at ttydev_read+0x55/frame 0xf8971b10 devfs_read_f(c93860a8,f8971be8,c93b1b00,0,ca244c40,...) at devfs_read_f+0xa8/frame 0xf8971b4c dofileread(c93860a8,f8971be8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf8971b80 kern_readv(ca244c40,0,f8971be8,c15abae0,bfbfedf7,...) at kern_readv+0x93/frame 0xf8971bc8 sys_read(ca244c40,f8971ca8,c11d8d0d,4,c1066357,...) at sys_read+0x59/frame 0xf8971c18 syscall(f8971ce8) at syscall+0x34a/frame 0xf8971cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8971cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command getty pid 887 tid 100146 td 0xc9e12620 sched_switch(c9e12620,0,104,1a9,f890b9a4,...) at sched_switch+0x4e2/frame 0xf890b96c mi_switch(104,0,c11e6431,200,c15ac2c8,...) at mi_switch+0x1bf/frame 0xf890b9a4 sleepq_switch(c9e12620,0,c11e6431,1b6,f890ba1c,...) at sleepq_switch+0x1c5/frame 0xf890b9d8 sleepq_catch_signals(f890ba38,c0b454f9,c8c78014,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf890ba14 sleepq_wait_sig(c8c78070,0,f890ba70,101,0,...) at sleepq_wait_sig+0x14/frame 0xf890ba38 _cv_wait_sig(c8c78070,c8c78004,c11eba4d,567,1,...) at _cv_wait_sig+0x214/frame 0xf890ba84 tty_wait(c8c78000,c8c78070,f890bbe8,1,0,...) at tty_wait+0x4b/frame 0xf890baa8 ttydisc_read(c8c78000,f890bbe8,0,b2,f890bbe8,...) at ttydisc_read+0x3d4/frame 0xf890baf0 ttydev_read(c8cc0900,f890bbe8,0,0,f890bb40,...) at ttydev_read+0x55/frame 0xf890bb10 devfs_read_f(c9426a10,f890bbe8,c93b1400,0,c9e12620,...) at devfs_read_f+0xa8/frame 0xf890bb4c dofileread(c9426a10,f890bbe8,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf890bb80 kern_readv(c9e12620,0,f890bbe8,c15ac2c8,bfbfedf7,...) at kern_readv+0x93/frame 0xf890bbc8 sys_read(c9e12620,f890bca8,c11d8d0d,4,c1066357,...) at sys_read+0x59/frame 0xf890bc18 syscall(f890bce8) at syscall+0x34a/frame 0xf890bcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf890bcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281b7f0f, esp = 0xbfbfedd8, ebp = 0xbfbfee08 --- Tracing command inetd pid 868 tid 100159 td 0xca246000 sched_switch(ca246000,0,104,1a9,f892f918,...) at sched_switch+0x4e2/frame 0xf892f8e0 mi_switch(104,0,c11e6431,200,f892f984,...) at mi_switch+0x1bf/frame 0xf892f918 sleepq_switch(ca246000,0,c11e6431,1b6,f892f990,...) at sleepq_switch+0x1c5/frame 0xf892f94c sleepq_catch_signals(f892f9ac,c0b454f9,c9475220,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf892f988 sleepq_wait_sig(c9475224,0,f892f9e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xf892f9ac _cv_wait_sig(c9475224,c9475210,c11e991b,732,c9ff9c78,...) at _cv_wait_sig+0x214/frame 0xf892f9f8 seltdwait(ffffffff,0,0,ca246000,0,...) at seltdwait+0xbb/frame 0xf892fa28 kern_select(ca246000,a,bfbfe078,0,0,...) at kern_select+0x914/frame 0xf892fbd0 sys_select(ca246000,f892fca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xf892fc18 syscall(f892fce8) at syscall+0x34a/frame 0xf892fcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf892fcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281cbeff, esp = 0xbfbfe014, ebp = 0xbfbfee34 --- Tracing command cron pid 689 tid 100103 td 0xc93d7310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c93d7310,ffffffff,2b1,18707,c93d7310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command sendmail pid 684 tid 100145 td 0xc9f78310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c9f78310,ffffffff,2ac,18731,c9f78310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command sendmail pid 681 tid 100143 td 0xc9f78930 sched_switch(c9f78930,0,104,1a9,f8902910,...) at sched_switch+0x4e2/frame 0xf89028d8 mi_switch(104,0,c11e6431,200,c1634210,...) at mi_switch+0x1bf/frame 0xf8902910 sleepq_switch(c9f78930,0,c11e6431,1b6,c93d5f50,...) at sleepq_switch+0x1c5/frame 0xf8902944 sleepq_catch_signals(c14559f0,f89029a8,c0b454f9,c93d5f60,0,...) at sleepq_catch_signals+0x382/frame 0xf8902980 sleepq_timedwait_sig(c93d5f64,0,f89029e4,50000000,0,...) at sleepq_timedwait_sig+0x14/frame 0xf89029a8 _cv_timedwait_sig_sbt(c93d5f64,c93d5f50,4b65bdf3,ad5c,50000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xf89029f8 seltdwait(ad5c,50000000,0,c9f78930,0,...) at seltdwait+0xad/frame 0xf8902a28 kern_select(c9f78930,5,bfbfd118,0,0,...) at kern_select+0x914/frame 0xf8902bd0 sys_select(c9f78930,f8902ca8,c11d8d0d,4,ca01c000,...) at sys_select+0x66/frame 0xf8902c18 syscall(f8902ce8) at syscall+0x34a/frame 0xf8902cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf8902cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28441eff, esp = 0xbfbfcfec, ebp = 0xbfbfdaa0 --- Tracing command sshd pid 678 tid 100095 td 0xc8efb000 sched_switch(c8efb000,0,104,1a9,edbc2918,...) at sched_switch+0x4e2/frame 0xedbc28e0 mi_switch(104,0,c11e6431,200,edbc2984,...) at mi_switch+0x1bf/frame 0xedbc2918 sleepq_switch(c8efb000,0,c11e6431,1b6,edbc2990,...) at sleepq_switch+0x1c5/frame 0xedbc294c sleepq_catch_signals(edbc29ac,c0b454f9,c93d5ee0,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbc2988 sleepq_wait_sig(c93d5ee4,0,edbc29e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbc29ac _cv_wait_sig(c93d5ee4,c93d5ed0,c11e991b,732,c9ffa720,...) at _cv_wait_sig+0x214/frame 0xedbc29f8 seltdwait(ffffffff,0,0,c8efb000,0,...) at seltdwait+0xbb/frame 0xedbc2a28 kern_select(c8efb000,7,28c0c158,0,0,...) at kern_select+0x914/frame 0xedbc2bd0 sys_select(c8efb000,edbc2ca8,c11d8d0d,4,8,...) at sys_select+0x66/frame 0xedbc2c18 syscall(edbc2ce8) at syscall+0x34a/frame 0xedbc2cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbc2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284e8eff, esp = 0xbfbfe88c, ebp = 0xbfbfe8b0 --- Tracing command moused pid 631 tid 100136 td 0xc9f79620 sched_switch(c9f79620,0,104,1a9,f88c4918,...) at sched_switch+0x4e2/frame 0xf88c48e0 mi_switch(104,0,c11e6431,200,246,...) at mi_switch+0x1bf/frame 0xf88c4918 sleepq_switch(c9f79620,0,c11e6431,1b6,f88c4990,...) at sleepq_switch+0x1c5/frame 0xf88c494c sleepq_catch_signals(f88c49ac,c0b454f9,c93d6160,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88c4988 sleepq_wait_sig(c93d6164,0,f88c49e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88c49ac _cv_wait_sig(c93d6164,c93d6150,c11e991b,732,1,...) at _cv_wait_sig+0x214/frame 0xf88c49f8 seltdwait(ffffffff,0,0,c9f79620,0,...) at seltdwait+0xbb/frame 0xf88c4a28 kern_select(c9f79620,400,bfbfeba0,0,0,...) at kern_select+0x914/frame 0xf88c4bd0 sys_select(c9f79620,f88c4ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xf88c4c18 syscall(f88c4ce8) at syscall+0x34a/frame 0xf88c4cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88c4cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281e2eff, esp = 0xbfbfea54, ebp = 0xbfbfee24 --- Tracing command smartd pid 624 tid 100137 td 0xc9f79310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c9f79310,ffffffff,270,18729,c9f79310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command powerd pid 605 tid 100138 td 0xc9f79000 sched_switch(c9f79000,0,104,1a9,f88d8910,...) at sched_switch+0x4e2/frame 0xf88d88d8 mi_switch(104,0,c11e6431,200,c1634090,...) at mi_switch+0x1bf/frame 0xf88d8910 sleepq_switch(c9f79000,0,c11e6431,1b6,c936d2d0,...) at sleepq_switch+0x1c5/frame 0xf88d8944 sleepq_catch_signals(c14559f0,f88d89a8,c0b454f9,c936d2e0,0,...) at sleepq_catch_signals+0x382/frame 0xf88d8980 sleepq_timedwait_sig(c936d2e4,0,f88d89e4,3ffffff,0,...) at sleepq_timedwait_sig+0x14/frame 0xf88d89a8 _cv_timedwait_sig_sbt(c936d2e4,c936d2d0,9d92d087,ad57,3ffffff,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xf88d89f8 seltdwait(ad57,3ffffff,0,f88d8a40,ffffffff,...) at seltdwait+0xad/frame 0xf88d8a28 kern_select(c9f79000,0,bfbfeda0,0,bfbfeda0,...) at kern_select+0x914/frame 0xf88d8bd0 sys_select(c9f79000,f88d8ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xf88d8c18 syscall(f88d8ce8) at syscall+0x34a/frame 0xf88d8cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xf88d8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281b5eff, esp = 0xbfbfed1c, ebp = 0xbfbfee44 --- Tracing command ntpd pid 602 tid 100093 td 0xc93d7c40 sched_switch(c93d7c40,0,104,1a9,edb9d664,...) at sched_switch+0x4e2/frame 0xedb9d62c mi_switch(104,0,c11e6431,200,c157d300,...) at mi_switch+0x1bf/frame 0xedb9d664 sleepq_switch(c93d7c40,0,c11e6431,276,c157d380,...) at sleepq_switch+0x1c5/frame 0xedb9d698 sleepq_wait(c157d380,0,edb9d6ec,1,0,...) at sleepq_wait+0x48/frame 0xedb9d6bc _cv_wait(c157d380,c157d300,c11e7a29,4b2,c121d821,...) at _cv_wait+0x1fe/frame 0xedb9d700 vmem_xalloc(c157d300,10000,0,0,0,...) at vmem_xalloc+0x1df/frame 0xedb9d774 vmem_alloc(c157d300,10000,2002,edb9d7c4,0,...) at vmem_alloc+0xc4/frame 0xedb9d7ac kmem_malloc(c157d300,10000,2) at kmem_malloc+0x3b/frame 0xedb9d7d4 page_alloc(c1f9a1e0,10000,edb9d817,2,c1634080,...) at page_alloc+0x28/frame 0xedb9d7e8 keg_alloc_slab(2,4,c121d821,92a,c1f9b200,...) at keg_alloc_slab+0xd2/frame 0xedb9d824 keg_fetch_slab(2,0,c121d821,975,c1f9a1e0,...) at keg_fetch_slab+0x18f/frame 0xedb9d868 zone_fetch_slab(c1f9a1e0,0,2,14,c11e2492,...) at zone_fetch_slab+0x5d/frame 0xedb9d88c zone_import(c1f9a1e0,edb9d8ec,1,2,0,...) at zone_import+0x37/frame 0xedb9d8d4 zone_alloc_item(2,d1534d40,2018,c11dbe14,1,...) at zone_alloc_item+0x39/frame 0xedb9d904 uma_zalloc_arg(c1f9a1e0,0,2,0,bfbfdd90,...) at uma_zalloc_arg+0x658/frame 0xedb9d944 malloc(10000,c1453dcc,2,0,0,...) at malloc+0x1b1/frame 0xedb9d990 sysctl_rtsock(c1461aa4,edb9dbb4,4,edb9db28,edb9dbb4,...) at sysctl_rtsock+0x106/frame 0xedb9daa8 sysctl_root_handler_locked(4,edb9db28,edb9dbb4,4,edb9db28,...) at sysctl_root_handler_locked+0xb2/frame 0xedb9dad0 sysctl_root(edb9db28,0,c11e0a40,705,0,...) at sysctl_root+0x19a/frame 0xedb9db08 userland_sysctl(c93d7c40,edb9dbac,6,0,bfbfe534,...) at userland_sysctl+0x19d/frame 0xedb9db70 sys___sysctl(c93d7c40,edb9dca8,c11d8d0d,4,c94a1000,...) at sys___sysctl+0x98/frame 0xedb9dc18 syscall(edb9dce8) at syscall+0x34a/frame 0xedb9dcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb9dcdc --- syscall (202, FreeBSD ELF32, sys___sysctl), eip = 0x2843f567, esp = 0xbfbfe4bc, ebp = 0xbfbfe4e4 --- Tracing command nfsd pid 549 tid 100094 td 0xc9476620 sched_switch(c9476620,0,104,1a9,edbbb310,...) at sched_switch+0x4e2/frame 0xedbbb2d8 mi_switch(104,0,c11e6431,200,c1634090,...) at mi_switch+0x1bf/frame 0xedbbb310 sleepq_switch(c9476620,0,c11e6431,1b6,c8e4f100,...) at sleepq_switch+0x1c5/frame 0xedbbb344 sleepq_catch_signals(c14559f0,edbbb3a8,c0b454f9,c8e4f110,0,...) at sleepq_catch_signals+0x382/frame 0xedbbb380 sleepq_timedwait_sig(c93b4190,0,edbbb3e4,0,0,...) at sleepq_timedwait_sig+0x14/frame 0xedbbb3a8 _cv_timedwait_sig_sbt(c93b4190,c8e4f100,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedbbb3f8 svc_run_internal(c0d93db0,c8e4f100,c946f9c0,edbbb514,0,...) at svc_run_internal+0xc42/frame 0xedbbb4dc svc_run(c8e4f000,0,c11ca37c,1e6,edbbb568,...) at svc_run+0x1bf/frame 0xedbbb524 nfsrvd_nfsd(c9476620,edbbb690,c,c11ce879,60,...) at nfsrvd_nfsd+0x1ba/frame 0xedbbb660 nfssvc_nfsd(c9476620,edbbbca8,edbbbca0,c9476620,edbbbcdc,...) at nfssvc_nfsd+0x11b/frame 0xedbbbc00 sys_nfssvc(c9476620,edbbbca8,c11d8d0d,4,c1066357,...) at sys_nfssvc+0xc4/frame 0xedbbbc18 syscall(edbbbce8) at syscall+0x34a/frame 0xedbbbcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbbbcdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280db1fb, esp = 0xbfbfe7e4, ebp = 0xbfbfea58 --- Tracing command nfsd pid 549 tid 100105 td 0xc9378c40 sched_switch(c9378c40,0,104,1a9,edbedae0,...) at sched_switch+0x4e2/frame 0xedbedaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbedae0 sleepq_switch(c9378c40,0,c11e6431,1b6,edbedb58,...) at sleepq_switch+0x1c5/frame 0xedbedb14 sleepq_catch_signals(edbedb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbedb50 sleepq_wait_sig(c9310b90,0,edbedbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbedb74 _cv_wait_sig(c9310b90,c8e4f100,c1211c05,414,0,...) at _cv_wait_sig+0x214/frame 0xedbedbc0 svc_run_internal(edbedcd4,c0b2780e,c8e4f100,edbedce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbedca4 svc_thread_start(c8e4f100,edbedce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbedcac fork_exit(c0d93db0,c8e4f100,edbedce8) at fork_exit+0x7e/frame 0xedbedcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbedcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100106 td 0xc9378930 sched_switch(c9378930,0,104,1a9,edbf0ae0,...) at sched_switch+0x4e2/frame 0xedbf0aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbf0ae0 sleepq_switch(c9378930,0,c11e6431,1b6,edbf0b58,...) at sleepq_switch+0x1c5/frame 0xedbf0b14 sleepq_catch_signals(edbf0b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbf0b50 sleepq_wait_sig(c89f5e90,0,edbf0bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbf0b74 _cv_wait_sig(c89f5e90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbf0bc0 svc_run_internal(edbf0cd4,c0b2780e,c8e4f100,edbf0ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbf0ca4 svc_thread_start(c8e4f100,edbf0ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbf0cac fork_exit(c0d93db0,c8e4f100,edbf0ce8) at fork_exit+0x7e/frame 0xedbf0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbf0cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100107 td 0xc9378620 sched_switch(c9378620,0,104,1a9,edbf3ae0,...) at sched_switch+0x4e2/frame 0xedbf3aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbf3ae0 sleepq_switch(c9378620,0,c11e6431,1b6,edbf3b58,...) at sleepq_switch+0x1c5/frame 0xedbf3b14 sleepq_catch_signals(edbf3b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbf3b50 sleepq_wait_sig(c9310a90,0,edbf3bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbf3b74 _cv_wait_sig(c9310a90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbf3bc0 svc_run_internal(edbf3cd4,c0b2780e,c8e4f100,edbf3ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbf3ca4 svc_thread_start(c8e4f100,edbf3ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbf3cac fork_exit(c0d93db0,c8e4f100,edbf3ce8) at fork_exit+0x7e/frame 0xedbf3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbf3cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100108 td 0xc9378310 sched_switch(c9378310,0,104,1a9,edbf6ae0,...) at sched_switch+0x4e2/frame 0xedbf6aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbf6ae0 sleepq_switch(c9378310,0,c11e6431,1b6,edbf6b58,...) at sleepq_switch+0x1c5/frame 0xedbf6b14 sleepq_catch_signals(edbf6b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbf6b50 sleepq_wait_sig(c89f5d90,0,edbf6bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbf6b74 _cv_wait_sig(c89f5d90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbf6bc0 svc_run_internal(edbf6cd4,c0b2780e,c8e4f100,edbf6ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbf6ca4 svc_thread_start(c8e4f100,edbf6ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbf6cac fork_exit(c0d93db0,c8e4f100,edbf6ce8) at fork_exit+0x7e/frame 0xedbf6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbf6cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100109 td 0xc9f74620 sched_switch(c9f74620,0,104,1a9,edbf9ae0,...) at sched_switch+0x4e2/frame 0xedbf9aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbf9ae0 sleepq_switch(c9f74620,0,c11e6431,1b6,edbf9b58,...) at sleepq_switch+0x1c5/frame 0xedbf9b14 sleepq_catch_signals(edbf9b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbf9b50 sleepq_wait_sig(c89f5c90,0,edbf9bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbf9b74 _cv_wait_sig(c89f5c90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbf9bc0 svc_run_internal(edbf9cd4,c0b2780e,c8e4f100,edbf9ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbf9ca4 svc_thread_start(c8e4f100,edbf9ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbf9cac fork_exit(c0d93db0,c8e4f100,edbf9ce8) at fork_exit+0x7e/frame 0xedbf9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbf9cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100110 td 0xc9f74310 sched_switch(c9f74310,0,104,1a9,edbfcae0,...) at sched_switch+0x4e2/frame 0xedbfcaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbfcae0 sleepq_switch(c9f74310,0,c11e6431,1b6,edbfcb58,...) at sleepq_switch+0x1c5/frame 0xedbfcb14 sleepq_catch_signals(edbfcb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbfcb50 sleepq_wait_sig(c9310990,0,edbfcbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbfcb74 _cv_wait_sig(c9310990,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbfcbc0 svc_run_internal(edbfccd4,c0b2780e,c8e4f100,edbfcce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbfcca4 svc_thread_start(c8e4f100,edbfcce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbfccac fork_exit(c0d93db0,c8e4f100,edbfcce8) at fork_exit+0x7e/frame 0xedbfccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbfccd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100111 td 0xc9f74000 sched_switch(c9f74000,0,104,1a9,edbffae0,...) at sched_switch+0x4e2/frame 0xedbffaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xedbffae0 sleepq_switch(c9f74000,0,c11e6431,1b6,edbffb58,...) at sleepq_switch+0x1c5/frame 0xedbffb14 sleepq_catch_signals(edbffb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbffb50 sleepq_wait_sig(c9310890,0,edbffbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbffb74 _cv_wait_sig(c9310890,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xedbffbc0 svc_run_internal(edbffcd4,c0b2780e,c8e4f100,edbffce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xedbffca4 svc_thread_start(c8e4f100,edbffce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xedbffcac fork_exit(c0d93db0,c8e4f100,edbffce8) at fork_exit+0x7e/frame 0xedbffcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedbffcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100112 td 0xc9f73c40 sched_switch(c9f73c40,0,104,1a9,f887cae0,...) at sched_switch+0x4e2/frame 0xf887caa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf887cae0 sleepq_switch(c9f73c40,0,c11e6431,1b6,f887cb58,...) at sleepq_switch+0x1c5/frame 0xf887cb14 sleepq_catch_signals(f887cb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf887cb50 sleepq_wait_sig(c89f5b90,0,f887cbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf887cb74 _cv_wait_sig(c89f5b90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf887cbc0 svc_run_internal(f887ccd4,c0b2780e,c8e4f100,f887cce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf887cca4 svc_thread_start(c8e4f100,f887cce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf887ccac fork_exit(c0d93db0,c8e4f100,f887cce8) at fork_exit+0x7e/frame 0xf887ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf887ccd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100113 td 0xc9f73930 sched_switch(c9f73930,0,104,1a9,f887fae0,...) at sched_switch+0x4e2/frame 0xf887faa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf887fae0 sleepq_switch(c9f73930,0,c11e6431,1b6,f887fb58,...) at sleepq_switch+0x1c5/frame 0xf887fb14 sleepq_catch_signals(f887fb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf887fb50 sleepq_wait_sig(c9310790,0,f887fbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf887fb74 _cv_wait_sig(c9310790,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf887fbc0 svc_run_internal(f887fcd4,c0b2780e,c8e4f100,f887fce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf887fca4 svc_thread_start(c8e4f100,f887fce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf887fcac fork_exit(c0d93db0,c8e4f100,f887fce8) at fork_exit+0x7e/frame 0xf887fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf887fcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100114 td 0xc9f73620 sched_switch(c9f73620,0,104,1a9,f8882ae0,...) at sched_switch+0x4e2/frame 0xf8882aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8882ae0 sleepq_switch(c9f73620,0,c11e6431,1b6,f8882b58,...) at sleepq_switch+0x1c5/frame 0xf8882b14 sleepq_catch_signals(f8882b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8882b50 sleepq_wait_sig(c89f5a90,0,f8882bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8882b74 _cv_wait_sig(c89f5a90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8882bc0 svc_run_internal(f8882cd4,c0b2780e,c8e4f100,f8882ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8882ca4 svc_thread_start(c8e4f100,f8882ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8882cac fork_exit(c0d93db0,c8e4f100,f8882ce8) at fork_exit+0x7e/frame 0xf8882cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8882cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100115 td 0xc9f73310 sched_switch(c9f73310,0,104,1a9,f8885ae0,...) at sched_switch+0x4e2/frame 0xf8885aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8885ae0 sleepq_switch(c9f73310,0,c11e6431,1b6,f8885b58,...) at sleepq_switch+0x1c5/frame 0xf8885b14 sleepq_catch_signals(f8885b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8885b50 sleepq_wait_sig(c9310690,0,f8885bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8885b74 _cv_wait_sig(c9310690,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8885bc0 svc_run_internal(f8885cd4,c0b2780e,c8e4f100,f8885ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8885ca4 svc_thread_start(c8e4f100,f8885ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8885cac fork_exit(c0d93db0,c8e4f100,f8885ce8) at fork_exit+0x7e/frame 0xf8885cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8885cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100116 td 0xc9f73000 sched_switch(c9f73000,0,104,1a9,f8888ae0,...) at sched_switch+0x4e2/frame 0xf8888aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8888ae0 sleepq_switch(c9f73000,0,c11e6431,1b6,f8888b58,...) at sleepq_switch+0x1c5/frame 0xf8888b14 sleepq_catch_signals(f8888b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8888b50 sleepq_wait_sig(c89f5990,0,f8888bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8888b74 _cv_wait_sig(c89f5990,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8888bc0 svc_run_internal(f8888cd4,c0b2780e,c8e4f100,f8888ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8888ca4 svc_thread_start(c8e4f100,f8888ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8888cac fork_exit(c0d93db0,c8e4f100,f8888ce8) at fork_exit+0x7e/frame 0xf8888cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8888cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100117 td 0xc9f72c40 sched_switch(c9f72c40,0,104,1a9,f888bae0,...) at sched_switch+0x4e2/frame 0xf888baa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf888bae0 sleepq_switch(c9f72c40,0,c11e6431,1b6,f888bb58,...) at sleepq_switch+0x1c5/frame 0xf888bb14 sleepq_catch_signals(f888bb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf888bb50 sleepq_wait_sig(c9310590,0,f888bbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf888bb74 _cv_wait_sig(c9310590,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf888bbc0 svc_run_internal(f888bcd4,c0b2780e,c8e4f100,f888bce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf888bca4 svc_thread_start(c8e4f100,f888bce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf888bcac fork_exit(c0d93db0,c8e4f100,f888bce8) at fork_exit+0x7e/frame 0xf888bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf888bcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100118 td 0xc9f72930 sched_switch(c9f72930,0,104,1a9,f888eae0,...) at sched_switch+0x4e2/frame 0xf888eaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf888eae0 sleepq_switch(c9f72930,0,c11e6431,1b6,f888eb58,...) at sleepq_switch+0x1c5/frame 0xf888eb14 sleepq_catch_signals(f888eb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf888eb50 sleepq_wait_sig(c89f5890,0,f888ebac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf888eb74 _cv_wait_sig(c89f5890,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf888ebc0 svc_run_internal(f888ecd4,c0b2780e,c8e4f100,f888ece8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf888eca4 svc_thread_start(c8e4f100,f888ece8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf888ecac fork_exit(c0d93db0,c8e4f100,f888ece8) at fork_exit+0x7e/frame 0xf888ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf888ecd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100119 td 0xc9f72620 sched_switch(c9f72620,0,104,1a9,f8891ae0,...) at sched_switch+0x4e2/frame 0xf8891aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8891ae0 sleepq_switch(c9f72620,0,c11e6431,1b6,f8891b58,...) at sleepq_switch+0x1c5/frame 0xf8891b14 sleepq_catch_signals(f8891b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8891b50 sleepq_wait_sig(c9310490,0,f8891bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8891b74 _cv_wait_sig(c9310490,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8891bc0 svc_run_internal(f8891cd4,c0b2780e,c8e4f100,f8891ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8891ca4 svc_thread_start(c8e4f100,f8891ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8891cac fork_exit(c0d93db0,c8e4f100,f8891ce8) at fork_exit+0x7e/frame 0xf8891cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8891cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100120 td 0xc9f72310 sched_switch(c9f72310,0,104,1a9,f8894ae0,...) at sched_switch+0x4e2/frame 0xf8894aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8894ae0 sleepq_switch(c9f72310,0,c11e6431,1b6,f8894b58,...) at sleepq_switch+0x1c5/frame 0xf8894b14 sleepq_catch_signals(f8894b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8894b50 sleepq_wait_sig(c89f5790,0,f8894bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8894b74 _cv_wait_sig(c89f5790,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8894bc0 svc_run_internal(f8894cd4,c0b2780e,c8e4f100,f8894ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8894ca4 svc_thread_start(c8e4f100,f8894ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8894cac fork_exit(c0d93db0,c8e4f100,f8894ce8) at fork_exit+0x7e/frame 0xf8894cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8894cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100121 td 0xc9f72000 sched_switch(c9f72000,0,104,1a9,f8897ae0,...) at sched_switch+0x4e2/frame 0xf8897aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf8897ae0 sleepq_switch(c9f72000,0,c11e6431,1b6,f8897b58,...) at sleepq_switch+0x1c5/frame 0xf8897b14 sleepq_catch_signals(f8897b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf8897b50 sleepq_wait_sig(c9310390,0,f8897bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf8897b74 _cv_wait_sig(c9310390,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf8897bc0 svc_run_internal(f8897cd4,c0b2780e,c8e4f100,f8897ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf8897ca4 svc_thread_start(c8e4f100,f8897ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf8897cac fork_exit(c0d93db0,c8e4f100,f8897ce8) at fork_exit+0x7e/frame 0xf8897cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8897cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100122 td 0xc9f77000 sched_switch(c9f77000,0,104,1a9,f889aae0,...) at sched_switch+0x4e2/frame 0xf889aaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf889aae0 sleepq_switch(c9f77000,0,c11e6431,1b6,f889ab58,...) at sleepq_switch+0x1c5/frame 0xf889ab14 sleepq_catch_signals(f889ab74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf889ab50 sleepq_wait_sig(c89f5690,0,f889abac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf889ab74 _cv_wait_sig(c89f5690,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf889abc0 svc_run_internal(f889acd4,c0b2780e,c8e4f100,f889ace8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf889aca4 svc_thread_start(c8e4f100,f889ace8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf889acac fork_exit(c0d93db0,c8e4f100,f889ace8) at fork_exit+0x7e/frame 0xf889acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf889acd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100123 td 0xc9f76c40 sched_switch(c9f76c40,0,104,1a9,f889dae0,...) at sched_switch+0x4e2/frame 0xf889daa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf889dae0 sleepq_switch(c9f76c40,0,c11e6431,1b6,f889db58,...) at sleepq_switch+0x1c5/frame 0xf889db14 sleepq_catch_signals(f889db74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf889db50 sleepq_wait_sig(c89f5590,0,f889dbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf889db74 _cv_wait_sig(c89f5590,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf889dbc0 svc_run_internal(f889dcd4,c0b2780e,c8e4f100,f889dce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf889dca4 svc_thread_start(c8e4f100,f889dce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf889dcac fork_exit(c0d93db0,c8e4f100,f889dce8) at fork_exit+0x7e/frame 0xf889dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf889dcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100124 td 0xc9f76930 sched_switch(c9f76930,0,104,1a9,f88a0ae0,...) at sched_switch+0x4e2/frame 0xf88a0aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88a0ae0 sleepq_switch(c9f76930,0,c11e6431,1b6,f88a0b58,...) at sleepq_switch+0x1c5/frame 0xf88a0b14 sleepq_catch_signals(f88a0b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88a0b50 sleepq_wait_sig(c89f5490,0,f88a0bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88a0b74 _cv_wait_sig(c89f5490,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88a0bc0 svc_run_internal(f88a0cd4,c0b2780e,c8e4f100,f88a0ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88a0ca4 svc_thread_start(c8e4f100,f88a0ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88a0cac fork_exit(c0d93db0,c8e4f100,f88a0ce8) at fork_exit+0x7e/frame 0xf88a0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88a0cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100125 td 0xc9f76620 sched_switch(c9f76620,0,104,1a9,f88a3ae0,...) at sched_switch+0x4e2/frame 0xf88a3aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88a3ae0 sleepq_switch(c9f76620,0,c11e6431,1b6,f88a3b58,...) at sleepq_switch+0x1c5/frame 0xf88a3b14 sleepq_catch_signals(f88a3b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88a3b50 sleepq_wait_sig(c9310290,0,f88a3bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88a3b74 _cv_wait_sig(c9310290,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88a3bc0 svc_run_internal(f88a3cd4,c0b2780e,c8e4f100,f88a3ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88a3ca4 svc_thread_start(c8e4f100,f88a3ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88a3cac fork_exit(c0d93db0,c8e4f100,f88a3ce8) at fork_exit+0x7e/frame 0xf88a3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88a3cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100126 td 0xc9f76310 sched_switch(c9f76310,0,104,1a9,f88a6ae0,...) at sched_switch+0x4e2/frame 0xf88a6aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88a6ae0 sleepq_switch(c9f76310,0,c11e6431,1b6,f88a6b58,...) at sleepq_switch+0x1c5/frame 0xf88a6b14 sleepq_catch_signals(f88a6b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88a6b50 sleepq_wait_sig(c9310190,0,f88a6bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88a6b74 _cv_wait_sig(c9310190,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88a6bc0 svc_run_internal(f88a6cd4,c0b2780e,c8e4f100,f88a6ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88a6ca4 svc_thread_start(c8e4f100,f88a6ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88a6cac fork_exit(c0d93db0,c8e4f100,f88a6ce8) at fork_exit+0x7e/frame 0xf88a6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88a6cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100127 td 0xc9f76000 sched_switch(c9f76000,0,104,1a9,f88a9ae0,...) at sched_switch+0x4e2/frame 0xf88a9aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88a9ae0 sleepq_switch(c9f76000,0,c11e6431,1b6,f88a9b58,...) at sleepq_switch+0x1c5/frame 0xf88a9b14 sleepq_catch_signals(f88a9b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88a9b50 sleepq_wait_sig(c89f5390,0,f88a9bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88a9b74 _cv_wait_sig(c89f5390,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88a9bc0 svc_run_internal(f88a9cd4,c0b2780e,c8e4f100,f88a9ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88a9ca4 svc_thread_start(c8e4f100,f88a9ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88a9cac fork_exit(c0d93db0,c8e4f100,f88a9ce8) at fork_exit+0x7e/frame 0xf88a9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88a9cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100128 td 0xc9f75c40 sched_switch(c9f75c40,0,104,1a9,f88acae0,...) at sched_switch+0x4e2/frame 0xf88acaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88acae0 sleepq_switch(c9f75c40,0,c11e6431,1b6,f88acb58,...) at sleepq_switch+0x1c5/frame 0xf88acb14 sleepq_catch_signals(f88acb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88acb50 sleepq_wait_sig(c9310090,0,f88acbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88acb74 _cv_wait_sig(c9310090,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88acbc0 svc_run_internal(f88accd4,c0b2780e,c8e4f100,f88acce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88acca4 svc_thread_start(c8e4f100,f88acce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88accac fork_exit(c0d93db0,c8e4f100,f88acce8) at fork_exit+0x7e/frame 0xf88accd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88accd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100129 td 0xc9f75930 sched_switch(c9f75930,0,104,1a9,f88afae0,...) at sched_switch+0x4e2/frame 0xf88afaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88afae0 sleepq_switch(c9f75930,0,c11e6431,1b6,f88afb58,...) at sleepq_switch+0x1c5/frame 0xf88afb14 sleepq_catch_signals(f88afb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88afb50 sleepq_wait_sig(c89f5290,0,f88afbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88afb74 _cv_wait_sig(c89f5290,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88afbc0 svc_run_internal(f88afcd4,c0b2780e,c8e4f100,f88afce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88afca4 svc_thread_start(c8e4f100,f88afce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88afcac fork_exit(c0d93db0,c8e4f100,f88afce8) at fork_exit+0x7e/frame 0xf88afcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88afcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100130 td 0xc9f75620 sched_switch(c9f75620,0,104,1a9,f88b2ae0,...) at sched_switch+0x4e2/frame 0xf88b2aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88b2ae0 sleepq_switch(c9f75620,0,c11e6431,1b6,f88b2b58,...) at sleepq_switch+0x1c5/frame 0xf88b2b14 sleepq_catch_signals(f88b2b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88b2b50 sleepq_wait_sig(c930fe90,0,f88b2bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88b2b74 _cv_wait_sig(c930fe90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88b2bc0 svc_run_internal(f88b2cd4,c0b2780e,c8e4f100,f88b2ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88b2ca4 svc_thread_start(c8e4f100,f88b2ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88b2cac fork_exit(c0d93db0,c8e4f100,f88b2ce8) at fork_exit+0x7e/frame 0xf88b2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88b2cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100131 td 0xc9f75310 sched_switch(c9f75310,0,104,1a9,f88b5ae0,...) at sched_switch+0x4e2/frame 0xf88b5aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88b5ae0 sleepq_switch(c9f75310,0,c11e6431,1b6,f88b5b58,...) at sleepq_switch+0x1c5/frame 0xf88b5b14 sleepq_catch_signals(f88b5b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88b5b50 sleepq_wait_sig(c930fd90,0,f88b5bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88b5b74 _cv_wait_sig(c930fd90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88b5bc0 svc_run_internal(f88b5cd4,c0b2780e,c8e4f100,f88b5ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88b5ca4 svc_thread_start(c8e4f100,f88b5ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88b5cac fork_exit(c0d93db0,c8e4f100,f88b5ce8) at fork_exit+0x7e/frame 0xf88b5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88b5cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100132 td 0xc9f75000 sched_switch(c9f75000,0,104,1a9,f88b8ae0,...) at sched_switch+0x4e2/frame 0xf88b8aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88b8ae0 sleepq_switch(c9f75000,0,c11e6431,1b6,f88b8b58,...) at sleepq_switch+0x1c5/frame 0xf88b8b14 sleepq_catch_signals(f88b8b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88b8b50 sleepq_wait_sig(c89f5190,0,f88b8bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88b8b74 _cv_wait_sig(c89f5190,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88b8bc0 svc_run_internal(f88b8cd4,c0b2780e,c8e4f100,f88b8ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88b8ca4 svc_thread_start(c8e4f100,f88b8ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88b8cac fork_exit(c0d93db0,c8e4f100,f88b8ce8) at fork_exit+0x7e/frame 0xf88b8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88b8cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100133 td 0xc9f74c40 sched_switch(c9f74c40,0,104,1a9,f88bbae0,...) at sched_switch+0x4e2/frame 0xf88bbaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88bbae0 sleepq_switch(c9f74c40,0,c11e6431,1b6,f88bbb58,...) at sleepq_switch+0x1c5/frame 0xf88bbb14 sleepq_catch_signals(f88bbb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88bbb50 sleepq_wait_sig(c930fc90,0,f88bbbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88bbb74 _cv_wait_sig(c930fc90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88bbbc0 svc_run_internal(f88bbcd4,c0b2780e,c8e4f100,f88bbce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88bbca4 svc_thread_start(c8e4f100,f88bbce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88bbcac fork_exit(c0d93db0,c8e4f100,f88bbce8) at fork_exit+0x7e/frame 0xf88bbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88bbcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100134 td 0xc9f74930 sched_switch(c9f74930,0,104,1a9,f88beae0,...) at sched_switch+0x4e2/frame 0xf88beaa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88beae0 sleepq_switch(c9f74930,0,c11e6431,1b6,f88beb58,...) at sleepq_switch+0x1c5/frame 0xf88beb14 sleepq_catch_signals(f88beb74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88beb50 sleepq_wait_sig(c930fb90,0,f88bebac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88beb74 _cv_wait_sig(c930fb90,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88bebc0 svc_run_internal(f88becd4,c0b2780e,c8e4f100,f88bece8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88beca4 svc_thread_start(c8e4f100,f88bece8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88becac fork_exit(c0d93db0,c8e4f100,f88bece8) at fork_exit+0x7e/frame 0xf88becd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88becd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 549 tid 100135 td 0xc9f79930 sched_switch(c9f79930,0,104,1a9,f88c1ae0,...) at sched_switch+0x4e2/frame 0xf88c1aa8 mi_switch(104,0,c11e6431,200,c15801dc,...) at mi_switch+0x1bf/frame 0xf88c1ae0 sleepq_switch(c9f79930,0,c11e6431,1b6,f88c1b58,...) at sleepq_switch+0x1c5/frame 0xf88c1b14 sleepq_catch_signals(f88c1b74,c0b454f9,c8e4f110,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xf88c1b50 sleepq_wait_sig(c89f5090,0,f88c1bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf88c1b74 _cv_wait_sig(c89f5090,c8e4f100,c1211c05,45c,0,...) at _cv_wait_sig+0x214/frame 0xf88c1bc0 svc_run_internal(f88c1cd4,c0b2780e,c8e4f100,f88c1ce8,c11d90ec,...) at svc_run_internal+0xc71/frame 0xf88c1ca4 svc_thread_start(c8e4f100,f88c1ce8,c11d90ec,3dc,0,...) at svc_thread_start+0xd/frame 0xf88c1cac fork_exit(c0d93db0,c8e4f100,f88c1ce8) at fork_exit+0x7e/frame 0xf88c1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf88c1cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 543 tid 100104 td 0xc93d7000 sched_switch(c93d7000,0,104,1a9,edbe4918,...) at sched_switch+0x4e2/frame 0xedbe48e0 mi_switch(104,0,c11e6431,200,edbe4984,...) at mi_switch+0x1bf/frame 0xedbe4918 sleepq_switch(c93d7000,0,c11e6431,1b6,edbe4990,...) at sleepq_switch+0x1c5/frame 0xedbe494c sleepq_catch_signals(edbe49ac,c0b454f9,c9475320,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbe4988 sleepq_wait_sig(c9475324,0,edbe49e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbe49ac _cv_wait_sig(c9475324,c9475310,c11e991b,732,c9ae4720,...) at _cv_wait_sig+0x214/frame 0xedbe49f8 seltdwait(ffffffff,0,0,c93d7000,0,...) at seltdwait+0xbb/frame 0xedbe4a28 kern_select(c93d7000,7,bfbfec30,0,0,...) at kern_select+0x914/frame 0xedbe4bd0 sys_select(c93d7000,edbe4ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xedbe4c18 syscall(edbe4ce8) at syscall+0x34a/frame 0xedbe4cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbe4cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a4eff, esp = 0xbfbfea5c, ebp = 0xbfbfee34 --- Tracing command mountd pid 537 tid 100097 td 0xc9e12930 sched_switch(c9e12930,0,104,1a9,edbc8918,...) at sched_switch+0x4e2/frame 0xedbc88e0 mi_switch(104,0,c11e6431,200,60,...) at mi_switch+0x1bf/frame 0xedbc8918 sleepq_switch(c9e12930,0,c11e6431,1b6,edbc8990,...) at sleepq_switch+0x1c5/frame 0xedbc894c sleepq_catch_signals(edbc89ac,c0b454f9,c93d6120,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbc8988 sleepq_wait_sig(c93d6124,0,edbc89e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbc89ac _cv_wait_sig(c93d6124,c93d6110,c11e991b,732,c9aef720,...) at _cv_wait_sig+0x214/frame 0xedbc89f8 seltdwait(ffffffff,0,0,c9e12930,0,...) at seltdwait+0xbb/frame 0xedbc8a28 kern_select(c9e12930,9,bfbfed50,0,0,...) at kern_select+0x914/frame 0xedbc8bd0 sys_select(c9e12930,edbc8ca8,c11d8d0d,4,8,...) at sys_select+0x66/frame 0xedbc8c18 syscall(edbc8ce8) at syscall+0x34a/frame 0xedbc8cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbc8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bbeff, esp = 0xbfbfed0c, ebp = 0xbfbfee34 --- Tracing command nfsuserd pid 530 tid 100102 td 0xc93d7620 sched_switch(c93d7620,0,104,1a9,edbdc910,...) at sched_switch+0x4e2/frame 0xedbdc8d8 mi_switch(104,0,c11e6431,200,c1634210,...) at mi_switch+0x1bf/frame 0xedbdc910 sleepq_switch(c93d7620,0,c11e6431,1b6,c936d350,...) at sleepq_switch+0x1c5/frame 0xedbdc944 sleepq_catch_signals(c14559f0,edbdc9a8,c0b454f9,c936d360,0,...) at sleepq_catch_signals+0x382/frame 0xedbdc980 sleepq_timedwait_sig(c936d364,0,edbdc9e4,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedbdc9a8 _cv_timedwait_sig_sbt(c936d364,c936d350,4f7e3c48,ad6e,e0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedbdc9f8 seltdwait(ad6e,e0000000,1,c93d7620,0,...) at seltdwait+0xad/frame 0xedbdca28 kern_select(c93d7620,4,bfbfec10,0,0,...) at kern_select+0x914/frame 0xedbdcbd0 sys_select(c93d7620,edbdcca8,c11d8d0d,4,c11e2492,...) at sys_select+0x66/frame 0xedbdcc18 syscall(edbdcce8) at syscall+0x34a/frame 0xedbdccdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbdccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a3eff, esp = 0xbfbfeb64, ebp = 0xbfbfeca8 --- Tracing command nfsuserd pid 529 tid 100101 td 0xc93d7930 sched_switch(c93d7930,0,104,1a9,edbd8910,...) at sched_switch+0x4e2/frame 0xedbd88d8 mi_switch(104,0,c11e6431,200,c1634210,...) at mi_switch+0x1bf/frame 0xedbd8910 sleepq_switch(c93d7930,0,c11e6431,1b6,c94755d0,...) at sleepq_switch+0x1c5/frame 0xedbd8944 sleepq_catch_signals(c14559f0,edbd89a8,c0b454f9,c94755e0,0,...) at sleepq_catch_signals+0x382/frame 0xedbd8980 sleepq_timedwait_sig(c94755e4,0,edbd89e4,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedbd89a8 _cv_timedwait_sig_sbt(c94755e4,c94755d0,43b74c45,ad5d,e0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedbd89f8 seltdwait(ad5d,e0000000,1,c93d7930,0,...) at seltdwait+0xad/frame 0xedbd8a28 kern_select(c93d7930,4,bfbfec10,0,0,...) at kern_select+0x914/frame 0xedbd8bd0 sys_select(c93d7930,edbd8ca8,c11d8d0d,4,c11e2492,...) at sys_select+0x66/frame 0xedbd8c18 syscall(edbd8ce8) at syscall+0x34a/frame 0xedbd8cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbd8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a3eff, esp = 0xbfbfeb64, ebp = 0xbfbfeca8 --- Tracing command nfsuserd pid 528 tid 100092 td 0xc93e4000 sched_switch(c93e4000,0,104,1a9,edb85910,...) at sched_switch+0x4e2/frame 0xedb858d8 mi_switch(104,0,c11e6431,200,c1634090,...) at mi_switch+0x1bf/frame 0xedb85910 sleepq_switch(c93e4000,0,c11e6431,1b6,c936d390,...) at sleepq_switch+0x1c5/frame 0xedb85944 sleepq_catch_signals(c14559f0,edb859a8,c0b454f9,c936d3a0,0,...) at sleepq_catch_signals+0x382/frame 0xedb85980 sleepq_timedwait_sig(c936d3a4,0,edb859e4,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedb859a8 _cv_timedwait_sig_sbt(c936d3a4,c936d390,4ae2647a,ad5c,e0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedb859f8 seltdwait(ad5c,e0000000,1,c93e4000,0,...) at seltdwait+0xad/frame 0xedb85a28 kern_select(c93e4000,4,bfbfec10,0,0,...) at kern_select+0x914/frame 0xedb85bd0 sys_select(c93e4000,edb85ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xedb85c18 syscall(edb85ce8) at syscall+0x34a/frame 0xedb85cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb85cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a3eff, esp = 0xbfbfeb64, ebp = 0xbfbfeca8 --- Tracing command nfsuserd pid 527 tid 100078 td 0xc9488310 sched_switch(c9488310,0,104,1a9,edb1b910,...) at sched_switch+0x4e2/frame 0xedb1b8d8 mi_switch(104,0,c11e6431,200,c1634090,...) at mi_switch+0x1bf/frame 0xedb1b910 sleepq_switch(c9488310,0,c11e6431,1b6,c9475690,...) at sleepq_switch+0x1c5/frame 0xedb1b944 sleepq_catch_signals(c14559f0,edb1b9a8,c0b454f9,c94756a0,0,...) at sleepq_catch_signals+0x382/frame 0xedb1b980 sleepq_timedwait_sig(c94756a4,0,edb1b9e4,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedb1b9a8 _cv_timedwait_sig_sbt(c94756a4,c9475690,5c39247,ad72,e0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedb1b9f8 seltdwait(ad72,e0000000,1,c9488310,0,...) at seltdwait+0xad/frame 0xedb1ba28 kern_select(c9488310,4,bfbfec10,0,0,...) at kern_select+0x914/frame 0xedb1bbd0 sys_select(c9488310,edb1bca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xedb1bc18 syscall(edb1bce8) at syscall+0x34a/frame 0xedb1bcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb1bcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a3eff, esp = 0xbfbfeb64, ebp = 0xbfbfeca8 --- Tracing command nfsuserd pid 526 tid 100100 td 0xc9377310 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(c11e4418,de9dd620,fa9f458c,8,0,...) at db_trace_self_wrapper+0x2a/frame 0xfa9f4570 kdb_backtrace(c11e4495,280,fa9f4724,c105dbd0,8,...) at kdb_backtrace+0x2d/frame 0xfa9f45d8 kdb_reenter(8,c11e6e52,225,fa9f4610,de9dd620,...) at kdb_reenter+0x31/frame 0xfa9f45e8 trap(fa9f4730) at trap+0x50/frame 0xfa9f4724 calltrap() at calltrap+0x6/frame 0xfa9f4724 --- trap 0xc, eip = 0xc10497f6, esp = 0xfa9f4770, ebp = 0xfa9f4780 --- db_trace_thread(c9377310,ffffffff,20e,18704,c9377310,...) at db_trace_thread+0x16/frame 0xfa9f4780 db_stack_trace_all(c163af08,0,ffffffff,fa9f4820,0,...) at db_stack_trace_all+0xa2/frame 0xfa9f47ec db_command(c1499602,0,c1629ae0,c1499601,a,...) at db_command+0x39f/frame 0xfa9f48a8 db_script_exec(fa9f4958,0,c8b58250,fa9f4944,1,...) at db_script_exec+0x168/frame 0xfa9f4910 db_command(c10ae563,fa9f4a30,c053337b,c10ae654,c079a0d7,...) at db_command+0x39f/frame 0xfa9f49cc db_command_loop(c10ae654,c079a0d7,c15de9a4,1,c8a12d00,...) at db_command_loop+0x50/frame 0xfa9f49d8 db_trap(3,0,c9f78930,46,1,...) at db_trap+0xdb/frame 0xfa9f4a30 kdb_trap(3,0,fa9f4bb0,fa9f4bf8,0,...) at kdb_trap+0x117/frame 0xfa9f4a68 trap(fa9f4bb0) at trap+0x87f/frame 0xfa9f4ba4 calltrap() at calltrap+0x6/frame 0xfa9f4ba4 --- trap 0x3, eip = 0xc0b9c832, esp = 0xfa9f4bf0, ebp = 0xfa9f4c04 --- kdb_alt_break_internal(0,fa9f4c30,c097dec7,2,c8cd7884,...) at kdb_alt_break_internal+0x162/frame 0xfa9f4c04 kdb_alt_break(2,c8cd7884,c11a8159,c8cd7800,40000,...) at kdb_alt_break+0x16/frame 0xfa9f4c10 uart_intr_rxready(c8cd7800,c13a3c53,df,ad57,1,...) at uart_intr_rxready+0x87/frame 0xfa9f4c30 uart_intr(c8cd7800,0,8,fa9f4cac,0,...) at uart_intr+0x100/frame 0xfa9f4c6c intr_event_handle(c88e2900,fa9f4ce8,62b0bfa7,2,0,...) at intr_event_handle+0x7a/frame 0xfa9f4c98 intr_execute_handlers(c8a90108,fa9f4ce8,2) at intr_execute_handlers+0x42/frame 0xfa9f4cb8 lapic_handle_intr(30,fa9f4ce8) at lapic_handle_intr+0x5d/frame 0xfa9f4cd8 Xapic_isr1() at Xapic_isr1+0x4a/frame 0xfa9f4cd8 --- interrupt, eip = 0x80495b0, esp = 0xbfbfe790, ebp = 0xbfbfe7a8 --- Tracing command casperd pid 509 tid 100099 td 0xc9377620 sched_switch(c9377620,0,104,1a9,edbd0918,...) at sched_switch+0x4e2/frame 0xedbd08e0 mi_switch(104,0,c11e6431,200,edbd0984,...) at mi_switch+0x1bf/frame 0xedbd0918 sleepq_switch(c9377620,0,c11e6431,1b6,edbd0990,...) at sleepq_switch+0x1c5/frame 0xedbd094c sleepq_catch_signals(edbd09ac,c0b454f9,c88e9e60,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbd0988 sleepq_wait_sig(c88e9e64,0,edbd09e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbd09ac _cv_wait_sig(c88e9e64,c88e9e50,c11e991b,732,c9b4a558,...) at _cv_wait_sig+0x214/frame 0xedbd09f8 seltdwait(ffffffff,0,0,c9377620,0,...) at seltdwait+0xbb/frame 0xedbd0a28 kern_select(c9377620,7,bfbfe9c8,0,0,...) at kern_select+0x914/frame 0xedbd0bd0 sys_select(c9377620,edbd0ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xedbd0c18 syscall(edbd0ce8) at syscall+0x34a/frame 0xedbd0cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbd0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281d1eff, esp = 0xbfbfe984, ebp = 0xbfbfee44 --- Tracing command casperd pid 508 tid 100098 td 0xc9377930 sched_switch(c9377930,0,104,1a9,edbcd918,...) at sched_switch+0x4e2/frame 0xedbcd8e0 mi_switch(104,0,c11e6431,200,edbcd984,...) at mi_switch+0x1bf/frame 0xedbcd918 sleepq_switch(c9377930,0,c11e6431,1b6,edbcd990,...) at sleepq_switch+0x1c5/frame 0xedbcd94c sleepq_catch_signals(edbcd9ac,c0b454f9,c93d64a0,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedbcd988 sleepq_wait_sig(c93d64a4,0,edbcd9e4,101,0,...) at sleepq_wait_sig+0x14/frame 0xedbcd9ac _cv_wait_sig(c93d64a4,c93d6490,c11e991b,732,c9b20390,...) at _cv_wait_sig+0x214/frame 0xedbcd9f8 seltdwait(ffffffff,0,0,c9377930,0,...) at seltdwait+0xbb/frame 0xedbcda28 kern_select(c9377930,6,bfbfe858,0,0,...) at kern_select+0x914/frame 0xedbcdbd0 sys_select(c9377930,edbcdca8,c11d8d0d,4,c156bd90,...) at sys_select+0x66/frame 0xedbcdc18 syscall(edbcdce8) at syscall+0x34a/frame 0xedbcdcdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedbcdcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281d1eff, esp = 0xbfbfe82c, ebp = 0xbfbfe8f0 --- Tracing command rpcbind pid 494 tid 100080 td 0xc9377c40 sched_switch(c9377c40,0,104,1a9,edb25948,...) at sched_switch+0x4e2/frame 0xedb25910 mi_switch(104,0,c11e6431,200,c1634210,...) at mi_switch+0x1bf/frame 0xedb25948 sleepq_switch(c9377c40,0,c11e6431,1b6,c93d64d0,...) at sleepq_switch+0x1c5/frame 0xedb2597c sleepq_catch_signals(c14559f0,edb259e0,c0b454f9,c93d64e0,0,...) at sleepq_catch_signals+0x382/frame 0xedb259b8 sleepq_timedwait_sig(c93d64e4,0,edb25a1c,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedb259e0 _cv_timedwait_sig_sbt(c93d64e4,c93d64d0,de155c24,ad67,e0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedb25a30 seltdwait(ad67,e0000000,1,c9377c40,0,...) at seltdwait+0xad/frame 0xedb25a60 kern_poll(c9377c40,bfbfccc0,7,edb25c00,0,...) at kern_poll+0x4f0/frame 0xedb25be0 sys_poll(c9377c40,edb25ca8,c11d8d0d,4,c11e2492,...) at sys_poll+0x71/frame 0xedb25c18 syscall(edb25ce8) at syscall+0x34a/frame 0xedb25cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb25cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x28150c53, esp = 0xbfbfcbfc, ebp = 0xbfbfedd8 --- Tracing command watchdogd pid 477 tid 100083 td 0xc9487c40 sched_switch(c9487c40,0,104,1a9,edb3a820,...) at sched_switch+0x4e2/frame 0xedb3a7e8 mi_switch(104,0,c11e6431,200,c157d300,...) at mi_switch+0x1bf/frame 0xedb3a820 sleepq_switch(c9487c40,0,c11e6431,276,c157d380,...) at sleepq_switch+0x1c5/frame 0xedb3a854 sleepq_wait(c157d380,0,edb3a8a8,1,0,...) at sleepq_wait+0x48/frame 0xedb3a878 _cv_wait(c157d380,c157d300,c11e7a29,4b2,c121d821,...) at _cv_wait+0x1fe/frame 0xedb3a8bc vmem_xalloc(c157d300,1000,0,0,0,...) at vmem_xalloc+0x1df/frame 0xedb3a930 vmem_alloc(c157d300,1000,2002,edb3a980,0,...) at vmem_alloc+0xc4/frame 0xedb3a968 kmem_malloc(c157d300,1000,2) at kmem_malloc+0x3b/frame 0xedb3a990 page_alloc(c1fa13c0,1000,edb3a9d3,2,529,...) at page_alloc+0x28/frame 0xedb3a9a4 keg_alloc_slab(102,4,c121d821,92a,c1fa0100,...) at keg_alloc_slab+0xd2/frame 0xedb3a9e0 keg_fetch_slab(102,0,c121d821,975,c1fa13c0,...) at keg_fetch_slab+0x18f/frame 0xedb3aa24 zone_fetch_slab(c1fa13c0,0,102,c15abda0,14,...) at zone_fetch_slab+0x5d/frame 0xedb3aa48 zone_import(c1fa13c0,edb3aaa8,1,102,0,...) at zone_import+0x37/frame 0xedb3aa90 zone_alloc_item(102,e0f4d040,2018,c10c439e,1,...) at zone_alloc_item+0x39/frame 0xedb3aac0 uma_zalloc_arg(c1fa13c0,0,102,c145202c,c145201c,...) at uma_zalloc_arg+0x658/frame 0xedb3ab00 malloc(abc,c1456164,102,c9470680,de831340,...) at malloc+0x1b1/frame 0xedb3ab4c sigacts_alloc(de8313f8,0,c11d90ec,1a0,c9470738,...) at sigacts_alloc+0x23/frame 0xedb3ab68 fork1(c9487c40,80000034,0,edb3ac10,0,...) at fork1+0xb6f/frame 0xedb3abf0 sys_vfork(c9487c40,edb3aca8,c11d8d0d,4,0,...) at sys_vfork+0x39/frame 0xedb3ac18 syscall(edb3ace8) at syscall+0x34a/frame 0xedb3acdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb3acdc --- syscall (66, FreeBSD ELF32, sys_vfork), eip = 0x28129d70, esp = 0xbfbfec70, ebp = 0xbfbfed08 --- Tracing command syslogd pid 473 tid 100082 td 0xc8efb310 sched_switch(c8efb310,0,104,1a9,edb36a14,...) at sched_switch+0x4e2/frame 0xedb369dc mi_switch(104,0,c11e6431,200,d2c99c00,...) at mi_switch+0x1bf/frame 0xedb36a14 sleepq_switch(c8efb310,0,c11e6431,276,c8efb310,...) at sleepq_switch+0x1c5/frame 0xedb36a48 sleepq_wait(d2c99c4c,5d,c11ef32d,0,0,...) at sleepq_wait+0x48/frame 0xedb36a6c _sleep(d2c99c4c,d2c99c00,5d,c11ef32d,0,...) at _sleep+0x3a1/frame 0xedb36ac8 bufobj_wwait(d2c99c00,0,0,0,0,...) at bufobj_wwait+0xb2/frame 0xedb36b04 ffs_syncvnode(d2c99b40,1,0,d0c04e14,c147ccf0,...) at ffs_syncvnode+0x3c6/frame 0xedb36b5c ffs_fsync(edb36bf0,c13acbb8,8,c11f33b4,da4,...) at ffs_fsync+0x26/frame 0xedb36b80 VOP_FSYNC_APV(c147ccf0,edb36bf0,da4,0,0,...) at VOP_FSYNC_APV+0x12f/frame 0xedb36bb0 sys_fsync(c8efb310,edb36ca8,c11d8d0d,4,c9ca3000,...) at sys_fsync+0x20d/frame 0xedb36c18 syscall(edb36ce8) at syscall+0x34a/frame 0xedb36cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb36cdc --- syscall (95, FreeBSD ELF32, sys_fsync), eip = 0x281b8bfb, esp = 0xbfbfdfdc, ebp = 0xbfbfee34 --- Tracing command rtsold pid 349 tid 100081 td 0xc9488000 sched_switch(c9488000,0,104,1a9,edb2c950,...) at sched_switch+0x4e2/frame 0xedb2c918 mi_switch(104,0,c11e6431,200,edb2c9bc,...) at mi_switch+0x1bf/frame 0xedb2c950 sleepq_switch(c9488000,0,c11e6431,1b6,edb2c9c8,...) at sleepq_switch+0x1c5/frame 0xedb2c984 sleepq_catch_signals(edb2c9e4,c0b454f9,c9475760,0,c14559f0,...) at sleepq_catch_signals+0x382/frame 0xedb2c9c0 sleepq_wait_sig(c9475764,0,edb2ca1c,101,0,...) at sleepq_wait_sig+0x14/frame 0xedb2c9e4 _cv_wait_sig(c9475764,c9475750,c11e991b,732,62d,...) at _cv_wait_sig+0x214/frame 0xedb2ca30 seltdwait(ffffffff,0,0,c9488000,0,...) at seltdwait+0xbb/frame 0xedb2ca60 kern_poll(c9488000,bfbfee08,2,0,0,...) at kern_poll+0x4f0/frame 0xedb2cbe0 sys_poll(c9488000,edb2cca8,c11d8d0d,4,c1066357,...) at sys_poll+0x71/frame 0xedb2cc18 syscall(edb2cce8) at syscall+0x34a/frame 0xedb2ccdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb2ccdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2814fc53, esp = 0xbfbfedcc, ebp = 0xbfbfee34 --- Tracing command devd pid 313 tid 100090 td 0xc9476c40 sched_switch(c9476c40,0,104,1a9,edb73910,...) at sched_switch+0x4e2/frame 0xedb738d8 mi_switch(104,0,c11e6431,200,c1633f10,...) at mi_switch+0x1bf/frame 0xedb73910 sleepq_switch(c9476c40,0,c11e6431,1b6,c94756d0,...) at sleepq_switch+0x1c5/frame 0xedb73944 sleepq_catch_signals(c14559f0,edb739a8,c0b454f9,c94756e0,0,...) at sleepq_catch_signals+0x382/frame 0xedb73980 sleepq_timedwait_sig(c94756e4,0,edb739e4,c0000000,3,...) at sleepq_timedwait_sig+0x14/frame 0xedb739a8 _cv_timedwait_sig_sbt(c94756e4,c94756d0,3eda8356,ad81,c0000000,...) at _cv_timedwait_sig_sbt+0x23c/frame 0xedb739f8 seltdwait(ad81,c0000000,3,c9476c40,0,...) at seltdwait+0xad/frame 0xedb73a28 kern_select(c9476c40,6,bfbfcda8,0,0,...) at kern_select+0x914/frame 0xedb73bd0 sys_select(c9476c40,edb73ca8,c11d8d0d,4,c1066357,...) at sys_select+0x66/frame 0xedb73c18 syscall(edb73ce8) at syscall+0x34a/frame 0xedb73cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xedb73cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x80e1bc3, esp = 0xbfbfcd4c, ebp = 0xbfbfee54 --- Tracing command vnlru pid 16 tid 100074 td 0xc8efbc40 sched_switch(c8efbc40,0,104,1a9,edb09b8c,...) at sched_switch+0x4e2/frame 0xedb09b54 mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb09b8c sleepq_switch(c8efbc40,0,c11e6431,299,c8efbc40,...) at sleepq_switch+0x1c5/frame 0xedb09bc0 sleepq_timedwait(c91749c0,60,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb09be4 _sleep(c91749c0,c15d838c,260,c11f2fc9,fffffed8,...) at _sleep+0x33d/frame 0xedb09c40 vnlru_proc(0,edb09ce8,c11d90ec,3dc,0,...) at vnlru_proc+0xcf/frame 0xedb09cac fork_exit(c0c1e890,0,edb09ce8) at fork_exit+0x7e/frame 0xedb09cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb09cd4 --- trap 0, eip = 0, esp = 0xedb09d20, ebp = 0 --- Tracing command syncer pid 9 tid 100073 td 0xc8efc000 sched_switch(c8efc000,0,104,1a9,edb068d0,...) at sched_switch+0x4e2/frame 0xedb06898 mi_switch(104,0,c11e6431,200,0,...) at mi_switch+0x1bf/frame 0xedb068d0 sleepq_switch(c8efc000,0,c11e6431,276,0,...) at sleepq_switch+0x1c5/frame 0xedb06904 sleepq_wait(e45a3da0,60,c11d6acd,4,0,...) at sleepq_wait+0x48/frame 0xedb06928 sleeplk(c931fed0,c11d6acd,60,0,0,...) at sleeplk+0x1a6/frame 0xedb06970 __lockmgr_args(e45a3da0,81900,c931fed0,c11d6acd,60,...) at __lockmgr_args+0xb74/frame 0xedb06a4c getblk(c931fe10,396e20,0,4000,0,...) at getblk+0x289/frame 0xedb06ac0 breadn_flags(c931fe10,396e20,0,4000,0,...) at breadn_flags+0x43/frame 0xedb06af0 ffs_update(d19b1b40,0,c8efc000,5a8,c11f1b68,...) at ffs_update+0x176/frame 0xedb06b60 ffs_sync(c930d2ec,3,2,c930d2ec,c145ff74,...) at ffs_sync+0x524/frame 0xedb06be4 sync_fsync(edb06c88,c13acbb8,80400,c11f1b68,6e9,...) at sync_fsync+0x116/frame 0xedb06c08 VOP_FSYNC_APV(c145ff74,edb06c88,c11f1b68,6e9,0,...) at VOP_FSYNC_APV+0x12f/frame 0xedb06c38 sched_sync(0,edb06ce8,c11d90ec,3dc,0,...) at sched_sync+0x441/frame 0xedb06cac fork_exit(c0c1e060,0,edb06ce8) at fork_exit+0x7e/frame 0xedb06cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb06cd4 --- trap 0, eip = 0, esp = 0xedb06d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100072 td 0xc8efc310 sched_switch(c8efc310,0,104,1a9,edb03bc4,...) at sched_switch+0x4e2/frame 0xedb03b8c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb03bc4 sleepq_switch(c8efc310,0,c11e6431,299,c8efc310,...) at sleepq_switch+0x1c5/frame 0xedb03bf8 sleepq_timedwait(c15d7b04,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb03c1c _sleep(c15d7b04,c15d7a80,54,c11ef8ae,fffffed8,...) at _sleep+0x33d/frame 0xedb03c78 buf_daemon(0,edb03ce8,c11d90ec,3dc,0,...) at buf_daemon+0x8c/frame 0xedb03cac fork_exit(c0c00ed0,0,edb03ce8) at fork_exit+0x7e/frame 0xedb03cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb03cd4 --- trap 0, eip = 0, esp = 0xedb03d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100084 td 0xc9487930 sched_switch(c9487930,0,104,1a9,edb3ebb4,...) at sched_switch+0x4e2/frame 0xedb3eb7c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb3ebb4 sleepq_switch(c9487930,0,c11e6431,299,c9487930,...) at sleepq_switch+0x1c5/frame 0xedb3ebe8 sleepq_timedwait(c96e4884,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb3ec0c _sleep(c96e4884,c96e4800,54,c121a968,7fffff6c,...) at _sleep+0x33d/frame 0xedb3ec68 softdep_flush(c9741000,edb3ece8,c11d90ec,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb3ecac fork_exit(c0dd8690,c9741000,edb3ece8) at fork_exit+0x7e/frame 0xedb3ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb3ecd4 --- trap 0, eip = 0, esp = 0xedb3ed20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100086 td 0xc9487310 sched_switch(c9487310,0,104,1a9,edb44bb4,...) at sched_switch+0x4e2/frame 0xedb44b7c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb44bb4 sleepq_switch(c9487310,0,c11e6431,299,c9487310,...) at sleepq_switch+0x1c5/frame 0xedb44be8 sleepq_timedwait(c96e4484,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb44c0c _sleep(c96e4484,c96e4400,54,c121a968,7fffff6c,...) at _sleep+0x33d/frame 0xedb44c68 softdep_flush(c97408c4,edb44ce8,c11d90ec,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb44cac fork_exit(c0dd8690,c97408c4,edb44ce8) at fork_exit+0x7e/frame 0xedb44cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb44cd4 --- trap 0, eip = 0, esp = 0xedb44d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100087 td 0xc9487000 sched_switch(c9487000,0,104,1a9,edb47bb4,...) at sched_switch+0x4e2/frame 0xedb47b7c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb47bb4 sleepq_switch(c9487000,0,c11e6431,299,c9487000,...) at sleepq_switch+0x1c5/frame 0xedb47be8 sleepq_timedwait(c96e4284,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedb47c0c _sleep(c96e4284,c96e4200,54,c121a968,7fffff6c,...) at _sleep+0x33d/frame 0xedb47c68 softdep_flush(c97405d8,edb47ce8,c11d90ec,3dc,0,...) at softdep_flush+0x1a2/frame 0xedb47cac fork_exit(c0dd8690,c97405d8,edb47ce8) at fork_exit+0x7e/frame 0xedb47cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb47cd4 --- trap 0, eip = 0, esp = 0xedb47d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100169 td 0xcab79620 sched_switch(cab79620,0,104,1a9,f8965bb4,...) at sched_switch+0x4e2/frame 0xf8965b7c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xf8965bb4 sleepq_switch(cab79620,0,c11e6431,299,cab79620,...) at sleepq_switch+0x1c5/frame 0xf8965be8 sleepq_timedwait(c92ca084,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf8965c0c _sleep(c92ca084,c92ca000,54,c121a968,7fffff6c,...) at _sleep+0x33d/frame 0xf8965c68 softdep_flush(c9740000,f8965ce8,c11d90ec,3dc,0,...) at softdep_flush+0x1a2/frame 0xf8965cac fork_exit(c0dd8690,c9740000,f8965ce8) at fork_exit+0x7e/frame 0xf8965cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf8965cd4 --- trap 0, eip = 0, esp = 0xf8965d20, ebp = 0 --- Tracing command bufdaemon pid 8 tid 101124 td 0xd9c45000 sched_switch(d9c45000,0,104,1a9,fb31ebb4,...) at sched_switch+0x4e2/frame 0xfb31eb7c mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xfb31ebb4 sleepq_switch(d9c45000,0,c11e6431,299,d9c45000,...) at sleepq_switch+0x1c5/frame 0xfb31ebe8 sleepq_timedwait(d2282484,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xfb31ec0c _sleep(d2282484,d2282400,54,c121a968,7fffff6c,...) at _sleep+0x33d/frame 0xfb31ec68 softdep_flush(cf614bb0,fb31ece8,c11d90ec,3dc,0,...) at softdep_flush+0x1a2/frame 0xfb31ecac fork_exit(c0dd8690,cf614bb0,fb31ece8) at fork_exit+0x7e/frame 0xfb31ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xfb31ecd4 --- trap 0, eip = 0, esp = 0xfb31ed20, ebp = 0 --- Tracing command pagezero pid 7 tid 100071 td 0xc8efc620 sched_switch(c8efc620,0,104,1a9,edb00bc0,...) at sched_switch+0x4e2/frame 0xedb00b88 mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xedb00bc0 sleepq_switch(c8efc620,0,c11e6431,299,c8efc620,...) at sleepq_switch+0x1c5/frame 0xedb00bf4 sleepq_timedwait(c15dd648,0,12b,0,0,...) at sleepq_timedwait+0x48/frame 0xedb00c18 _sleep(c15dd648,c163ae80,0,c122295d,fffea520,...) at _sleep+0x33d/frame 0xedb00c74 vm_pagezero(0,edb00ce8,c11d90ec,3dc,0,...) at vm_pagezero+0xc2/frame 0xedb00cac fork_exit(c0e3a130,0,edb00ce8) at fork_exit+0x7e/frame 0xedb00cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedb00cd4 --- trap 0, eip = 0, esp = 0xedb00d20, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100070 td 0xc8efc930 sched_switch(c8efc930,0,104,1a9,edafdb8c,...) at sched_switch+0x4e2/frame 0xedafdb54 mi_switch(104,0,c11e6431,200,c15dd3ac,...) at mi_switch+0x1bf/frame 0xedafdb8c sleepq_switch(c8efc930,0,c11e6431,276,c8efc930,...) at sleepq_switch+0x1c5/frame 0xedafdbc0 sleepq_wait(c15dd3c4,74,c11ef8ae,0,0,...) at sleepq_wait+0x48/frame 0xedafdbe4 _sleep(c15dd3c4,c15dd3ac,74,c11ef8ae,0,...) at _sleep+0x3a1/frame 0xedafdc40 vm_daemon(0,edafdce8,c11d90ec,3dc,0,...) at vm_daemon+0x9c/frame 0xedafdcac fork_exit(c0e316d0,0,edafdce8) at fork_exit+0x7e/frame 0xedafdcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedafdcd4 --- trap 0, eip = 0, esp = 0xedafdd20, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100069 td 0xc8efcc40 sched_switch(c8efcc40,0,104,1a9,edafa9e4,...) at sched_switch+0x4e2/frame 0xedafa9ac mi_switch(104,0,c11e6431,200,c163b000,...) at mi_switch+0x1bf/frame 0xedafa9e4 sleepq_switch(c8efcc40,0,c11e6431,276,c8efcc40,...) at sleepq_switch+0x1c5/frame 0xedafaa18 sleepq_wait(c15dcc24,54,c1221515,0,0,...) at sleepq_wait+0x48/frame 0xedafaa3c _sleep(c15dcc24,c163b000,54,c1221515,0,...) at _sleep+0x3a1/frame 0xedafaa98 getpbuf(c15dcc24,0,c121d0b9,565,8,...) at getpbuf+0xe8/frame 0xedafaacc swap_pager_putpages(d2434444,edafabc8,1,0,edafab38,...) at swap_pager_putpages+0x20c/frame 0xedafab1c vm_pageout_flush(edafabc8,1,0,0,0,0) at vm_pageout_flush+0xfb/frame 0xedafab68 vm_pageout_cluster(d2434454,4,c122107d,3a1,7fffff6c,...) at vm_pageout_cluster+0x3d1/frame 0xedafac40 vm_pageout(0,edaface8,c11d90ec,3dc,0,...) at vm_pageout+0xcf2/frame 0xedafacac fork_exit(c0e31ef0,0,edaface8) at fork_exit+0x7e/frame 0xedafacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedafacd4 --- trap 0, eip = 0, esp = 0xedafad20, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100066 td 0xc8efd620 sched_switch(c8efd620,0,104,1a9,ed971bcc,...) at sched_switch+0x4e2/frame 0xed971b94 mi_switch(104,0,c11e6431,200,c1637864,...) at mi_switch+0x1bf/frame 0xed971bcc sleepq_switch(c8efd620,0,c11e6431,276,c8efd620,...) at sleepq_switch+0x1c5/frame 0xed971c00 sleepq_wait(c163789c,0,c1206905,0,0,...) at sleepq_wait+0x48/frame 0xed971c24 _sleep(c163789c,c1637864,0,c1206905,0,...) at _sleep+0x3a1/frame 0xed971c80 sctp_iterator_thread(0,ed971ce8,c11d90ec,3dc,0,...) at sctp_iterator_thread+0x7c/frame 0xed971cac fork_exit(c0cc5cb0,0,ed971ce8) at fork_exit+0x7e/frame 0xed971cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed971cd4 --- trap 0, eip = 0, esp = 0xed971d20, ebp = 0 --- Tracing command fdc0 pid 3 tid 100061 td 0xc8ca4310 sched_switch(c8ca4310,0,104,1a9,ed961b60,...) at sched_switch+0x4e2/frame 0xed961b28 mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xed961b60 sleepq_switch(c8ca4310,0,c11e6431,299,c8ca4310,...) at sleepq_switch+0x1c5/frame 0xed961b94 sleepq_timedwait(c8c2ca3c,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xed961bb8 _sleep(c8c2ca3c,c8c2caf0,5c,c11cafb6,fffffed8,...) at _sleep+0x33d/frame 0xed961c14 fdc_thread(c8c2ca00,ed961ce8,c11d90ec,3dc,0,...) at fdc_thread+0x95a/frame 0xed961cac fork_exit(c0fe05a0,c8c2ca00,ed961ce8) at fork_exit+0x7e/frame 0xed961cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed961cd4 --- trap 0, eip = 0, esp = 0xed961d20, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc8be5930 sched_switch(c8be5930,0,104,1a9,ed8e8be8,...) at sched_switch+0x4e2/frame 0xed8e8bb0 mi_switch(104,0,c11e6431,200,c8c3ecc8,...) at mi_switch+0x1bf/frame 0xed8e8be8 sleepq_switch(c8be5930,0,c11e6431,276,c8c3eacc,...) at sleepq_switch+0x1c5/frame 0xed8e8c1c sleepq_wait(c8c3eacc,0,ed8e8c70,1,0,...) at sleepq_wait+0x48/frame 0xed8e8c40 _cv_wait(c8c3eacc,c8c3ecc8,c11b124b,76,c8c3ead4,...) at _cv_wait+0x1fe/frame 0xed8e8c84 usb_process(c8c3eac4,ed8e8ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8e8cac fork_exit(c09b9f30,c8c3eac4,ed8e8ce8) at fork_exit+0x7e/frame 0xed8e8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8e8cd4 --- trap 0, eip = 0, esp = 0xed8e8d20, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc8be5620 sched_switch(c8be5620,0,104,1a9,ed8ebbe8,...) at sched_switch+0x4e2/frame 0xed8ebbb0 mi_switch(104,0,c11e6431,200,c8c3ecc8,...) at mi_switch+0x1bf/frame 0xed8ebbe8 sleepq_switch(c8be5620,0,c11e6431,276,c8c3eafc,...) at sleepq_switch+0x1c5/frame 0xed8ebc1c sleepq_wait(c8c3eafc,0,ed8ebc70,1,0,...) at sleepq_wait+0x48/frame 0xed8ebc40 _cv_wait(c8c3eafc,c8c3ecc8,c11b124b,76,c8c3eb04,...) at _cv_wait+0x1fe/frame 0xed8ebc84 usb_process(c8c3eaf4,ed8ebce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8ebcac fork_exit(c09b9f30,c8c3eaf4,ed8ebce8) at fork_exit+0x7e/frame 0xed8ebcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8ebcd4 --- trap 0, eip = 0, esp = 0xed8ebd20, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc8be5310 sched_switch(c8be5310,0,104,1a9,ed8eebe8,...) at sched_switch+0x4e2/frame 0xed8eebb0 mi_switch(104,0,c11e6431,200,c8c3ecc8,...) at mi_switch+0x1bf/frame 0xed8eebe8 sleepq_switch(c8be5310,0,c11e6431,276,c8c3eb2c,...) at sleepq_switch+0x1c5/frame 0xed8eec1c sleepq_wait(c8c3eb2c,0,ed8eec70,1,0,...) at sleepq_wait+0x48/frame 0xed8eec40 _cv_wait(c8c3eb2c,c8c3ecc8,c11b124b,76,c8c3eb34,...) at _cv_wait+0x1fe/frame 0xed8eec84 usb_process(c8c3eb24,ed8eece8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8eecac fork_exit(c09b9f30,c8c3eb24,ed8eece8) at fork_exit+0x7e/frame 0xed8eecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8eecd4 --- trap 0, eip = 0, esp = 0xed8eed20, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc8be5000 sched_switch(c8be5000,0,104,1a9,ed8f1be8,...) at sched_switch+0x4e2/frame 0xed8f1bb0 mi_switch(104,0,c11e6431,200,c8c3ecc8,...) at mi_switch+0x1bf/frame 0xed8f1be8 sleepq_switch(c8be5000,0,c11e6431,276,c8c3eb5c,...) at sleepq_switch+0x1c5/frame 0xed8f1c1c sleepq_wait(c8c3eb5c,0,ed8f1c70,1,0,...) at sleepq_wait+0x48/frame 0xed8f1c40 _cv_wait(c8c3eb5c,c8c3ecc8,c11b124b,76,c8c3eb64,...) at _cv_wait+0x1fe/frame 0xed8f1c84 usb_process(c8c3eb54,ed8f1ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8f1cac fork_exit(c09b9f30,c8c3eb54,ed8f1ce8) at fork_exit+0x7e/frame 0xed8f1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f1cd4 --- trap 0, eip = 0, esp = 0xed8f1d20, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc8afc930 sched_switch(c8afc930,0,104,1a9,ed8f8be8,...) at sched_switch+0x4e2/frame 0xed8f8bb0 mi_switch(104,0,c11e6431,200,c8c8bcc8,...) at mi_switch+0x1bf/frame 0xed8f8be8 sleepq_switch(c8afc930,0,c11e6431,276,c8c8bacc,...) at sleepq_switch+0x1c5/frame 0xed8f8c1c sleepq_wait(c8c8bacc,0,ed8f8c70,1,0,...) at sleepq_wait+0x48/frame 0xed8f8c40 _cv_wait(c8c8bacc,c8c8bcc8,c11b124b,76,c8c8bad4,...) at _cv_wait+0x1fe/frame 0xed8f8c84 usb_process(c8c8bac4,ed8f8ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8f8cac fork_exit(c09b9f30,c8c8bac4,ed8f8ce8) at fork_exit+0x7e/frame 0xed8f8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f8cd4 --- trap 0, eip = 0, esp = 0xed8f8d20, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc8afc620 sched_switch(c8afc620,0,104,1a9,ed8fbbe8,...) at sched_switch+0x4e2/frame 0xed8fbbb0 mi_switch(104,0,c11e6431,200,c8c8bcc8,...) at mi_switch+0x1bf/frame 0xed8fbbe8 sleepq_switch(c8afc620,0,c11e6431,276,c8c8bafc,...) at sleepq_switch+0x1c5/frame 0xed8fbc1c sleepq_wait(c8c8bafc,0,ed8fbc70,1,0,...) at sleepq_wait+0x48/frame 0xed8fbc40 _cv_wait(c8c8bafc,c8c8bcc8,c11b124b,76,c8c8bb04,...) at _cv_wait+0x1fe/frame 0xed8fbc84 usb_process(c8c8baf4,ed8fbce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8fbcac fork_exit(c09b9f30,c8c8baf4,ed8fbce8) at fork_exit+0x7e/frame 0xed8fbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8fbcd4 --- trap 0, eip = 0, esp = 0xed8fbd20, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc8afc310 sched_switch(c8afc310,0,104,1a9,ed8febe8,...) at sched_switch+0x4e2/frame 0xed8febb0 mi_switch(104,0,c11e6431,200,c8c8bcc8,...) at mi_switch+0x1bf/frame 0xed8febe8 sleepq_switch(c8afc310,0,c11e6431,276,c8c8bb2c,...) at sleepq_switch+0x1c5/frame 0xed8fec1c sleepq_wait(c8c8bb2c,0,ed8fec70,1,0,...) at sleepq_wait+0x48/frame 0xed8fec40 _cv_wait(c8c8bb2c,c8c8bcc8,c11b124b,76,c8c8bb34,...) at _cv_wait+0x1fe/frame 0xed8fec84 usb_process(c8c8bb24,ed8fece8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed8fecac fork_exit(c09b9f30,c8c8bb24,ed8fece8) at fork_exit+0x7e/frame 0xed8fecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8fecd4 --- trap 0, eip = 0, esp = 0xed8fed20, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc8c93310 sched_switch(c8c93310,0,104,1a9,ed901be8,...) at sched_switch+0x4e2/frame 0xed901bb0 mi_switch(104,0,c11e6431,200,c8c8bcc8,...) at mi_switch+0x1bf/frame 0xed901be8 sleepq_switch(c8c93310,0,c11e6431,276,c8c8bb5c,...) at sleepq_switch+0x1c5/frame 0xed901c1c sleepq_wait(c8c8bb5c,0,ed901c70,1,0,...) at sleepq_wait+0x48/frame 0xed901c40 _cv_wait(c8c8bb5c,c8c8bcc8,c11b124b,76,c8c8bb64,...) at _cv_wait+0x1fe/frame 0xed901c84 usb_process(c8c8bb54,ed901ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed901cac fork_exit(c09b9f30,c8c8bb54,ed901ce8) at fork_exit+0x7e/frame 0xed901cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed901cd4 --- trap 0, eip = 0, esp = 0xed901d20, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc8c92c40 sched_switch(c8c92c40,0,104,1a9,ed908be8,...) at sched_switch+0x4e2/frame 0xed908bb0 mi_switch(104,0,c11e6431,200,c8c94cc8,...) at mi_switch+0x1bf/frame 0xed908be8 sleepq_switch(c8c92c40,0,c11e6431,276,c8c94acc,...) at sleepq_switch+0x1c5/frame 0xed908c1c sleepq_wait(c8c94acc,0,ed908c70,1,0,...) at sleepq_wait+0x48/frame 0xed908c40 _cv_wait(c8c94acc,c8c94cc8,c11b124b,76,c8c94ad4,...) at _cv_wait+0x1fe/frame 0xed908c84 usb_process(c8c94ac4,ed908ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed908cac fork_exit(c09b9f30,c8c94ac4,ed908ce8) at fork_exit+0x7e/frame 0xed908cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed908cd4 --- trap 0, eip = 0, esp = 0xed908d20, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc8c92930 sched_switch(c8c92930,0,104,1a9,ed90bbe8,...) at sched_switch+0x4e2/frame 0xed90bbb0 mi_switch(104,0,c11e6431,200,c8c94cc8,...) at mi_switch+0x1bf/frame 0xed90bbe8 sleepq_switch(c8c92930,0,c11e6431,276,c8c94afc,...) at sleepq_switch+0x1c5/frame 0xed90bc1c sleepq_wait(c8c94afc,0,ed90bc70,1,0,...) at sleepq_wait+0x48/frame 0xed90bc40 _cv_wait(c8c94afc,c8c94cc8,c11b124b,76,c8c94b04,...) at _cv_wait+0x1fe/frame 0xed90bc84 usb_process(c8c94af4,ed90bce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed90bcac fork_exit(c09b9f30,c8c94af4,ed90bce8) at fork_exit+0x7e/frame 0xed90bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90bcd4 --- trap 0, eip = 0, esp = 0xed90bd20, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc8c92620 sched_switch(c8c92620,0,104,1a9,ed90ebe8,...) at sched_switch+0x4e2/frame 0xed90ebb0 mi_switch(104,0,c11e6431,200,c8c94cc8,...) at mi_switch+0x1bf/frame 0xed90ebe8 sleepq_switch(c8c92620,0,c11e6431,276,c8c94b2c,...) at sleepq_switch+0x1c5/frame 0xed90ec1c sleepq_wait(c8c94b2c,0,ed90ec70,1,0,...) at sleepq_wait+0x48/frame 0xed90ec40 _cv_wait(c8c94b2c,c8c94cc8,c11b124b,76,c8c94b34,...) at _cv_wait+0x1fe/frame 0xed90ec84 usb_process(c8c94b24,ed90ece8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed90ecac fork_exit(c09b9f30,c8c94b24,ed90ece8) at fork_exit+0x7e/frame 0xed90ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed90ecd4 --- trap 0, eip = 0, esp = 0xed90ed20, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc8c92310 sched_switch(c8c92310,0,104,1a9,ed911be8,...) at sched_switch+0x4e2/frame 0xed911bb0 mi_switch(104,0,c11e6431,200,c8c94cc8,...) at mi_switch+0x1bf/frame 0xed911be8 sleepq_switch(c8c92310,0,c11e6431,276,c8c94b5c,...) at sleepq_switch+0x1c5/frame 0xed911c1c sleepq_wait(c8c94b5c,0,ed911c70,1,0,...) at sleepq_wait+0x48/frame 0xed911c40 _cv_wait(c8c94b5c,c8c94cc8,c11b124b,76,c8c94b64,...) at _cv_wait+0x1fe/frame 0xed911c84 usb_process(c8c94b54,ed911ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed911cac fork_exit(c09b9f30,c8c94b54,ed911ce8) at fork_exit+0x7e/frame 0xed911cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed911cd4 --- trap 0, eip = 0, esp = 0xed911d20, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc8c92000 sched_switch(c8c92000,0,104,1a9,ed915be8,...) at sched_switch+0x4e2/frame 0xed915bb0 mi_switch(104,0,c11e6431,200,c8c9acc8,...) at mi_switch+0x1bf/frame 0xed915be8 sleepq_switch(c8c92000,0,c11e6431,276,c8c9aacc,...) at sleepq_switch+0x1c5/frame 0xed915c1c sleepq_wait(c8c9aacc,0,ed915c70,1,0,...) at sleepq_wait+0x48/frame 0xed915c40 _cv_wait(c8c9aacc,c8c9acc8,c11b124b,76,c8c9aad4,...) at _cv_wait+0x1fe/frame 0xed915c84 usb_process(c8c9aac4,ed915ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed915cac fork_exit(c09b9f30,c8c9aac4,ed915ce8) at fork_exit+0x7e/frame 0xed915cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed915cd4 --- trap 0, eip = 0, esp = 0xed915d20, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc8c91c40 sched_switch(c8c91c40,0,104,1a9,ed918be8,...) at sched_switch+0x4e2/frame 0xed918bb0 mi_switch(104,0,c11e6431,200,c8c9acc8,...) at mi_switch+0x1bf/frame 0xed918be8 sleepq_switch(c8c91c40,0,c11e6431,276,c8c9aafc,...) at sleepq_switch+0x1c5/frame 0xed918c1c sleepq_wait(c8c9aafc,0,ed918c70,1,0,...) at sleepq_wait+0x48/frame 0xed918c40 _cv_wait(c8c9aafc,c8c9acc8,c11b124b,76,c8c9ab04,...) at _cv_wait+0x1fe/frame 0xed918c84 usb_process(c8c9aaf4,ed918ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed918cac fork_exit(c09b9f30,c8c9aaf4,ed918ce8) at fork_exit+0x7e/frame 0xed918cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed918cd4 --- trap 0, eip = 0, esp = 0xed918d20, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc8c91930 sched_switch(c8c91930,0,104,1a9,ed91bbe8,...) at sched_switch+0x4e2/frame 0xed91bbb0 mi_switch(104,0,c11e6431,200,c8c9acc8,...) at mi_switch+0x1bf/frame 0xed91bbe8 sleepq_switch(c8c91930,0,c11e6431,276,c8c9ab2c,...) at sleepq_switch+0x1c5/frame 0xed91bc1c sleepq_wait(c8c9ab2c,0,ed91bc70,1,0,...) at sleepq_wait+0x48/frame 0xed91bc40 _cv_wait(c8c9ab2c,c8c9acc8,c11b124b,76,c8c9ab34,...) at _cv_wait+0x1fe/frame 0xed91bc84 usb_process(c8c9ab24,ed91bce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed91bcac fork_exit(c09b9f30,c8c9ab24,ed91bce8) at fork_exit+0x7e/frame 0xed91bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91bcd4 --- trap 0, eip = 0, esp = 0xed91bd20, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc8c91620 sched_switch(c8c91620,0,104,1a9,ed91ebe8,...) at sched_switch+0x4e2/frame 0xed91ebb0 mi_switch(104,0,c11e6431,200,c8c9acc8,...) at mi_switch+0x1bf/frame 0xed91ebe8 sleepq_switch(c8c91620,0,c11e6431,276,c8c9ab5c,...) at sleepq_switch+0x1c5/frame 0xed91ec1c sleepq_wait(c8c9ab5c,0,ed91ec70,1,0,...) at sleepq_wait+0x48/frame 0xed91ec40 _cv_wait(c8c9ab5c,c8c9acc8,c11b124b,76,c8c9ab64,...) at _cv_wait+0x1fe/frame 0xed91ec84 usb_process(c8c9ab54,ed91ece8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed91ecac fork_exit(c09b9f30,c8c9ab54,ed91ece8) at fork_exit+0x7e/frame 0xed91ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed91ecd4 --- trap 0, eip = 0, esp = 0xed91ed20, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc8c91310 sched_switch(c8c91310,0,104,1a9,ed922be8,...) at sched_switch+0x4e2/frame 0xed922bb0 mi_switch(104,0,c11e6431,200,c8ca2cc8,...) at mi_switch+0x1bf/frame 0xed922be8 sleepq_switch(c8c91310,0,c11e6431,276,c8ca2acc,...) at sleepq_switch+0x1c5/frame 0xed922c1c sleepq_wait(c8ca2acc,0,ed922c70,1,0,...) at sleepq_wait+0x48/frame 0xed922c40 _cv_wait(c8ca2acc,c8ca2cc8,c11b124b,76,c8ca2ad4,...) at _cv_wait+0x1fe/frame 0xed922c84 usb_process(c8ca2ac4,ed922ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed922cac fork_exit(c09b9f30,c8ca2ac4,ed922ce8) at fork_exit+0x7e/frame 0xed922cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed922cd4 --- trap 0, eip = 0, esp = 0xed922d20, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc8c91000 sched_switch(c8c91000,0,104,1a9,ed925be8,...) at sched_switch+0x4e2/frame 0xed925bb0 mi_switch(104,0,c11e6431,200,c8ca2cc8,...) at mi_switch+0x1bf/frame 0xed925be8 sleepq_switch(c8c91000,0,c11e6431,276,c8ca2afc,...) at sleepq_switch+0x1c5/frame 0xed925c1c sleepq_wait(c8ca2afc,0,ed925c70,1,0,...) at sleepq_wait+0x48/frame 0xed925c40 _cv_wait(c8ca2afc,c8ca2cc8,c11b124b,76,c8ca2b04,...) at _cv_wait+0x1fe/frame 0xed925c84 usb_process(c8ca2af4,ed925ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed925cac fork_exit(c09b9f30,c8ca2af4,ed925ce8) at fork_exit+0x7e/frame 0xed925cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed925cd4 --- trap 0, eip = 0, esp = 0xed925d20, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc8be6c40 sched_switch(c8be6c40,0,104,1a9,ed928be8,...) at sched_switch+0x4e2/frame 0xed928bb0 mi_switch(104,0,c11e6431,200,c8ca2cc8,...) at mi_switch+0x1bf/frame 0xed928be8 sleepq_switch(c8be6c40,0,c11e6431,276,c8ca2b2c,...) at sleepq_switch+0x1c5/frame 0xed928c1c sleepq_wait(c8ca2b2c,0,ed928c70,1,0,...) at sleepq_wait+0x48/frame 0xed928c40 _cv_wait(c8ca2b2c,c8ca2cc8,c11b124b,76,c8ca2b34,...) at _cv_wait+0x1fe/frame 0xed928c84 usb_process(c8ca2b24,ed928ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed928cac fork_exit(c09b9f30,c8ca2b24,ed928ce8) at fork_exit+0x7e/frame 0xed928cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed928cd4 --- trap 0, eip = 0, esp = 0xed928d20, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc8ca5c40 sched_switch(c8ca5c40,0,104,1a9,ed92bbe8,...) at sched_switch+0x4e2/frame 0xed92bbb0 mi_switch(104,0,c11e6431,200,c8ca2cc8,...) at mi_switch+0x1bf/frame 0xed92bbe8 sleepq_switch(c8ca5c40,0,c11e6431,276,c8ca2b5c,...) at sleepq_switch+0x1c5/frame 0xed92bc1c sleepq_wait(c8ca2b5c,0,ed92bc70,1,0,...) at sleepq_wait+0x48/frame 0xed92bc40 _cv_wait(c8ca2b5c,c8ca2cc8,c11b124b,76,c8ca2b64,...) at _cv_wait+0x1fe/frame 0xed92bc84 usb_process(c8ca2b54,ed92bce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed92bcac fork_exit(c09b9f30,c8ca2b54,ed92bce8) at fork_exit+0x7e/frame 0xed92bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed92bcd4 --- trap 0, eip = 0, esp = 0xed92bd20, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc8ca5620 sched_switch(c8ca5620,0,104,1a9,ed934be8,...) at sched_switch+0x4e2/frame 0xed934bb0 mi_switch(104,0,c11e6431,200,c8c44ea0,...) at mi_switch+0x1bf/frame 0xed934be8 sleepq_switch(c8ca5620,0,c11e6431,276,c8c44ca4,...) at sleepq_switch+0x1c5/frame 0xed934c1c sleepq_wait(c8c44ca4,0,ed934c70,1,0,...) at sleepq_wait+0x48/frame 0xed934c40 _cv_wait(c8c44ca4,c8c44ea0,c11b124b,76,c8c44cac,...) at _cv_wait+0x1fe/frame 0xed934c84 usb_process(c8c44c9c,ed934ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed934cac fork_exit(c09b9f30,c8c44c9c,ed934ce8) at fork_exit+0x7e/frame 0xed934cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed934cd4 --- trap 0, eip = 0, esp = 0xed934d20, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc8ca5310 sched_switch(c8ca5310,0,104,1a9,ed937be8,...) at sched_switch+0x4e2/frame 0xed937bb0 mi_switch(104,0,c11e6431,200,c8c44ea0,...) at mi_switch+0x1bf/frame 0xed937be8 sleepq_switch(c8ca5310,0,c11e6431,276,c8c44cd4,...) at sleepq_switch+0x1c5/frame 0xed937c1c sleepq_wait(c8c44cd4,0,ed937c70,1,0,...) at sleepq_wait+0x48/frame 0xed937c40 _cv_wait(c8c44cd4,c8c44ea0,c11b124b,76,c8c44cdc,...) at _cv_wait+0x1fe/frame 0xed937c84 usb_process(c8c44ccc,ed937ce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed937cac fork_exit(c09b9f30,c8c44ccc,ed937ce8) at fork_exit+0x7e/frame 0xed937cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed937cd4 --- trap 0, eip = 0, esp = 0xed937d20, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc8ca5000 sched_switch(c8ca5000,0,104,1a9,ed93abe8,...) at sched_switch+0x4e2/frame 0xed93abb0 mi_switch(104,0,c11e6431,200,c8c44ea0,...) at mi_switch+0x1bf/frame 0xed93abe8 sleepq_switch(c8ca5000,0,c11e6431,276,c8c44d04,...) at sleepq_switch+0x1c5/frame 0xed93ac1c sleepq_wait(c8c44d04,0,ed93ac70,1,0,...) at sleepq_wait+0x48/frame 0xed93ac40 _cv_wait(c8c44d04,c8c44ea0,c11b124b,76,c8c44d0c,...) at _cv_wait+0x1fe/frame 0xed93ac84 usb_process(c8c44cfc,ed93ace8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed93acac fork_exit(c09b9f30,c8c44cfc,ed93ace8) at fork_exit+0x7e/frame 0xed93acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93acd4 --- trap 0, eip = 0, esp = 0xed93ad20, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc8ca4c40 sched_switch(c8ca4c40,0,104,1a9,ed93dbe8,...) at sched_switch+0x4e2/frame 0xed93dbb0 mi_switch(104,0,c11e6431,200,c8c44ea0,...) at mi_switch+0x1bf/frame 0xed93dbe8 sleepq_switch(c8ca4c40,0,c11e6431,276,c8c44d34,...) at sleepq_switch+0x1c5/frame 0xed93dc1c sleepq_wait(c8c44d34,0,ed93dc70,1,0,...) at sleepq_wait+0x48/frame 0xed93dc40 _cv_wait(c8c44d34,c8c44ea0,c11b124b,76,c8c44d3c,...) at _cv_wait+0x1fe/frame 0xed93dc84 usb_process(c8c44d2c,ed93dce8,c11d90ec,3dc,0,...) at usb_process+0x140/frame 0xed93dcac fork_exit(c09b9f30,c8c44d2c,ed93dce8) at fork_exit+0x7e/frame 0xed93dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed93dcd4 --- trap 0, eip = 0, esp = 0xed93dd20, ebp = 0 --- Tracing command cam pid 2 tid 100024 td 0xc8afa000 sched_switch(c8afa000,0,104,1a9,e477bba4,...) at sched_switch+0x4e2/frame 0xe477bb6c mi_switch(104,0,c11e6431,200,c1495b80,...) at mi_switch+0x1bf/frame 0xe477bba4 sleepq_switch(c8afa000,0,c11e6431,276,c8afa000,...) at sleepq_switch+0x1c5/frame 0xe477bbd8 sleepq_wait(c1495c00,5c,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe477bbfc _sleep(c1495c00,c1495b80,5c,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe477bc58 xpt_done_td(c1495b80,e477bce8,c11d90ec,3dc,0,...) at xpt_done_td+0x17e/frame 0xe477bcac fork_exit(c04c7fa0,c1495b80,e477bce8) at fork_exit+0x7e/frame 0xe477bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe477bcd4 --- trap 0, eip = 0, esp = 0xe477bd20, ebp = 0 --- Tracing command cam pid 2 tid 100068 td 0xc8efd000 sched_switch(c8efd000,0,104,1a9,ed977ba4,...) at sched_switch+0x4e2/frame 0xed977b6c mi_switch(104,0,c11e6431,200,c1495afc,...) at mi_switch+0x1bf/frame 0xed977ba4 sleepq_switch(c8efd000,0,c11e6431,276,c8efd000,...) at sleepq_switch+0x1c5/frame 0xed977bd8 sleepq_wait(c1495aac,5c,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xed977bfc _sleep(c1495aac,c1495afc,5c,c11cafb6,0,...) at _sleep+0x3a1/frame 0xed977c58 xpt_scanner_thread(0,ed977ce8,c11d90ec,3dc,0,...) at xpt_scanner_thread+0xa2/frame 0xed977cac fork_exit(c04c8360,0,ed977ce8) at fork_exit+0x7e/frame 0xed977cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed977cd4 --- trap 0, eip = 0, esp = 0xed977d20, ebp = 0 --- Tracing command rand_harvestq pid 14 tid 100017 td 0xc8afb620 sched_switch(c8afb620,0,104,1a9,e4766bc8,...) at sched_switch+0x4e2/frame 0xe4766b90 mi_switch(104,0,c11e6431,200,5,...) at mi_switch+0x1bf/frame 0xe4766bc8 sleepq_switch(c8afb620,0,c11e6431,299,c8afb620,...) at sleepq_switch+0x1c5/frame 0xe4766bfc sleepq_timedwait(c14b07d0,0,0,0,0,...) at sleepq_timedwait+0x48/frame 0xe4766c20 _sleep(c14b07d0,0,0,c11cafb6,19999999,...) at _sleep+0x33d/frame 0xe4766c7c random_kthread(0,e4766ce8,c11d90ec,3dc,0,...) at random_kthread+0x5c/frame 0xe4766cac fork_exit(c0898ce0,0,e4766ce8) at fork_exit+0x7e/frame 0xe4766cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4766cd4 --- trap 0, eip = 0, esp = 0xe4766d20, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc8a2e000 sched_switch(c8a2e000,0,104,1a9,e475abbc,...) at sched_switch+0x4e2/frame 0xe475ab84 mi_switch(104,0,c11e6431,200,c155c6bc,...) at mi_switch+0x1bf/frame 0xe475abbc sleepq_switch(c8a2e000,0,c11e6431,276,c8a2e000,...) at sleepq_switch+0x1c5/frame 0xe475abf0 sleepq_wait(c1632a20,5c,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe475ac14 _sleep(c1632a20,c155c6bc,25c,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe475ac70 g_run_events(0,e475ace8,c11d90ec,3dc,0,...) at g_run_events+0x67/frame 0xe475acac fork_exit(c0ab1730,0,e475ace8) at fork_exit+0x7e/frame 0xe475acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe475acd4 --- trap 0, eip = 0, esp = 0xe475ad20, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc8afc000 sched_switch(c8afc000,0,104,1a9,e475dbac,...) at sched_switch+0x4e2/frame 0xe475db74 mi_switch(104,0,c11e6431,200,c155c704,...) at mi_switch+0x1bf/frame 0xe475dbac sleepq_switch(c8afc000,0,c11e6431,276,c8afc000,...) at sleepq_switch+0x1c5/frame 0xe475dbe0 sleepq_wait(c1632a24,5c,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe475dc04 _sleep(c1632a24,c155c704,25c,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe475dc60 g_io_schedule_up(c8afc000,0,c11ccd8d,60,e475dcd4,...) at g_io_schedule_up+0x12a/frame 0xe475dc94 g_up_procbody(0,e475dce8,c11d90ec,3dc,0,...) at g_up_procbody+0x9d/frame 0xe475dcac fork_exit(c0ab17d0,0,e475dce8) at fork_exit+0x7e/frame 0xe475dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe475dcd4 --- trap 0, eip = 0, esp = 0xe475dd20, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc8afbc40 sched_switch(c8afbc40,0,104,1a9,e4760ba8,...) at sched_switch+0x4e2/frame 0xe4760b70 mi_switch(104,0,c11e6431,200,c155c6e4,...) at mi_switch+0x1bf/frame 0xe4760ba8 sleepq_switch(c8afbc40,0,c11e6431,276,c8afbc40,...) at sleepq_switch+0x1c5/frame 0xe4760bdc sleepq_wait(c1632a28,5c,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe4760c00 _sleep(c1632a28,c155c6e4,25c,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe4760c5c g_io_schedule_down(c8afbc40,0,c11ccd8d,6d,e4760cd4,...) at g_io_schedule_down+0x16c/frame 0xe4760c94 g_down_procbody(0,e4760ce8,c11d90ec,3dc,0,...) at g_down_procbody+0x9d/frame 0xe4760cac fork_exit(c0ab1870,0,e4760ce8) at fork_exit+0x7e/frame 0xe4760cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4760cd4 --- trap 0, eip = 0, esp = 0xe4760d20, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc8a39310 sched_switch(c8a39310,0,109,1a9,c1633f00,...) at sched_switch+0x4e2/frame 0xe4748c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xe4748c74 ithread_loop(c8a92020,e4748ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4748cac fork_exit(c0b2a720,c8a92020,e4748ce8) at fork_exit+0x7e/frame 0xe4748cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4748cd4 --- trap 0, eip = 0, esp = 0xe4748d20, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc8a39000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xc8a2ec40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc8a2e930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc8a2e620 sched_switch(c8a2e620,0,109,1a9,c15ac218,...) at sched_switch+0x4e2/frame 0xe4754c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xe4754c74 ithread_loop(c8a91f80,e4754ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4754cac fork_exit(c0b2a720,c8a91f80,e4754ce8) at fork_exit+0x7e/frame 0xe4754cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4754cd4 --- trap 0, eip = 0, esp = 0xe4754d20, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc8a2e310 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xc8afb000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100025 td 0xc8a3ac40 sched_switch(c8a3ac40,0,109,1a9,c15abf58,...) at sched_switch+0x4e2/frame 0xe477ec3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xe477ec74 ithread_loop(c8a91f50,e477ece8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xe477ecac fork_exit(c0b2a720,c8a91f50,e477ece8) at fork_exit+0x7e/frame 0xe477ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe477ecd4 --- trap 0, eip = 0, esp = 0xe477ed20, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc8a3a930 sched_switch(c8a3a930,0,109,1a9,c16333d0,...) at sched_switch+0x4e2/frame 0xe4781c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xe4781c74 ithread_loop(c8a91f40,e4781ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xe4781cac fork_exit(c0b2a720,c8a91f40,e4781ce8) at fork_exit+0x7e/frame 0xe4781cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4781cd4 --- trap 0, eip = 0, esp = 0xe4781d20, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc8be6620 sched_switch(c8be6620,0,109,1a9,c15ad558,...) at sched_switch+0x4e2/frame 0xe47b6c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xe47b6c74 ithread_loop(c8a91f20,e47b6ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xe47b6cac fork_exit(c0b2a720,c8a91f20,e47b6ce8) at fork_exit+0x7e/frame 0xe47b6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe47b6cd4 --- trap 0, eip = 0, esp = 0xe47b6d20, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc8be6310 sched_switch(c8be6310,0,109,1a9,c8c2e000,...) at sched_switch+0x4e2/frame 0xf0ebfc3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xf0ebfc74 ithread_loop(c8a91ef0,f0ebfce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xf0ebfcac fork_exit(c0b2a720,c8a91ef0,f0ebfce8) at fork_exit+0x7e/frame 0xf0ebfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ebfcd4 --- trap 0, eip = 0, esp = 0xf0ebfd20, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc8be6000 sched_switch(c8be6000,0,109,1a9,c1644934,...) at sched_switch+0x4e2/frame 0xf0ec3c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xf0ec3c74 ithread_loop(c8a91ee0,f0ec3ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xf0ec3cac fork_exit(c0b2a720,c8a91ee0,f0ec3ce8) at fork_exit+0x7e/frame 0xf0ec3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf0ec3cd4 --- trap 0, eip = 0, esp = 0xf0ec3d20, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc8be5c40 sched_switch(c8be5c40,0,109,1a9,c15ad500,...) at sched_switch+0x4e2/frame 0xed8e5c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xed8e5c74 ithread_loop(c8a91ed0,ed8e5ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xed8e5cac fork_exit(c0b2a720,c8a91ed0,ed8e5ce8) at fork_exit+0x7e/frame 0xed8e5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8e5cd4 --- trap 0, eip = 0, esp = 0xed8e5d20, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc8afcc40 sched_switch(c8afcc40,0,109,1a9,c15ac8a0,...) at sched_switch+0x4e2/frame 0xed8f5c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xed8f5c74 ithread_loop(c8a91ec0,ed8f5ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xed8f5cac fork_exit(c0b2a720,c8a91ec0,ed8f5ce8) at fork_exit+0x7e/frame 0xed8f5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed8f5cd4 --- trap 0, eip = 0, esp = 0xed8f5d20, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc8c93000 sched_switch(c8c93000,0,109,1a9,c15ac848,...) at sched_switch+0x4e2/frame 0xed905c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xed905c74 ithread_loop(c8a91eb0,ed905ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xed905cac fork_exit(c0b2a720,c8a91eb0,ed905ce8) at fork_exit+0x7e/frame 0xed905cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed905cd4 --- trap 0, eip = 0, esp = 0xed905d20, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc8ca5930 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xc8ca4930 sched_switch(c8ca4930,0,109,1a9,c1644934,...) at sched_switch+0x4e2/frame 0xed943c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xed943c74 ithread_loop(c8a91e90,ed943ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xed943cac fork_exit(c0b2a720,c8a91e90,ed943ce8) at fork_exit+0x7e/frame 0xed943cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed943cd4 --- trap 0, eip = 0, esp = 0xed943d20, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc8ca4620 sched_switch(c8ca4620,0,109,1a9,c16333d0,...) at sched_switch+0x4e2/frame 0xed954c3c mi_switch(109,0,c11d935d,544,8,...) at mi_switch+0x1bf/frame 0xed954c74 ithread_loop(c8a91e80,ed954ce8,c11d90ec,3dc,0,...) at ithread_loop+0x1e1/frame 0xed954cac fork_exit(c0b2a720,c8a91e80,ed954ce8) at fork_exit+0x7e/frame 0xed954cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed954cd4 --- trap 0, eip = 0, esp = 0xed954d20, ebp = 0 --- Tracing command intr pid 12 tid 100062 td 0xc8ca4000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc8c93c40 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc8a3a000 sched_switch(c8a3a000,0,608,1a4,ffffffff,...) at sched_switch+0x4e2/frame 0xe473bb6c mi_switch(608,0,c11e0290,d3,c8a3a000,...) at mi_switch+0x1bf/frame 0xe473bba4 critical_exit(c1628188,0,0,0) at critical_exit+0xa2/frame 0xe473bbc0 lapic_handle_timer(e473bbec) at lapic_handle_timer+0x90/frame 0xe473bbe0 Xtimerint() at Xtimerint+0x20/frame 0xe473bbe0 --- interrupt, eip = 0xc104fb67, esp = 0xe473bc2c, ebp = 0xe473bc30 --- spinlock_exit(c157c038,e473bc80,c11e2492,a48,c156be08,...) at spinlock_exit+0x27/frame 0xe473bc30 sched_idletd(0,e473bce8,c11d90ec,3dc,0,...) at sched_idletd+0x333/frame 0xe473bcac fork_exit(c0b8a2a0,0,e473bce8) at fork_exit+0x7e/frame 0xe473bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe473bcd4 --- trap 0, eip = 0, esp = 0xe473bd20, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc8a39c40 sched_switch(c8a39c40,0,108,1a4,8,...) at sched_switch+0x4e2/frame 0xe473ebf8 mi_switch(108,0,c11e2492,405,c156c508,...) at mi_switch+0x1bf/frame 0xe473ec30 sched_idletd(0,e473ece8,c11d90ec,3dc,0,...) at sched_idletd+0x42d/frame 0xe473ecac fork_exit(c0b8a2a0,0,e473ece8) at fork_exit+0x7e/frame 0xe473ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe473ecd4 --- trap 0, eip = 0, esp = 0xe473ed20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc8a39930 sched_switch(c8a39930,0,108,1a4,8,...) at sched_switch+0x4e2/frame 0xe4741bf8 mi_switch(108,0,c11e2492,405,c156cc08,...) at mi_switch+0x1bf/frame 0xe4741c30 sched_idletd(0,e4741ce8,c11d90ec,3dc,0,...) at sched_idletd+0x42d/frame 0xe4741cac fork_exit(c0b8a2a0,0,e4741ce8) at fork_exit+0x7e/frame 0xe4741cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4741cd4 --- trap 0, eip = 0, esp = 0xe4741d20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc8a39620 sched_switch(c8a39620,0,108,1a4,8,...) at sched_switch+0x4e2/frame 0xe4744bf8 mi_switch(108,0,c11e2492,405,c156d308,...) at mi_switch+0x1bf/frame 0xe4744c30 sched_idletd(0,e4744ce8,c11d90ec,3dc,0,...) at sched_idletd+0x42d/frame 0xe4744cac fork_exit(c0b8a2a0,0,e4744ce8) at fork_exit+0x7e/frame 0xe4744cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4744cd4 --- trap 0, eip = 0, esp = 0xe4744d20, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc8a3a310 sched_switch(c8a3a310,0,104,1a9,e47379a0,...) at sched_switch+0x4e2/frame 0xe4737968 mi_switch(104,0,c11e6431,200,c88eaf1c,...) at mi_switch+0x1bf/frame 0xe47379a0 sleepq_switch(c8a3a310,0,c11e6431,1b6,c11e6431,...) at sleepq_switch+0x1c5/frame 0xe47379d4 sleepq_catch_signals(c11e6431,16d,c8a2ca78,d8,c8a3a310,...) at sleepq_catch_signals+0x382/frame 0xe4737a10 sleepq_wait_sig(c8a2c9c0,6c,c11ea1c9,100,0,...) at sleepq_wait_sig+0x14/frame 0xe4737a34 _sleep(c8a2c9c0,c8a2ca68,16c,c11ea1c9,0,...) at _sleep+0x356/frame 0xe4737a90 kern_wait6(c8a3a310,7,0,0,e4737b24,...) at kern_wait6+0x45b/frame 0xe4737af8 sys_wait4(c8a3a310,e4737ca8,c11d8d0d,4,7b5,...) at sys_wait4+0x8b/frame 0xe4737c18 syscall(e4737ce8) at syscall+0x34a/frame 0xe4737cdc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe4737cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80653ef, esp = 0xbfbfe904, ebp = 0xbfbfe918 --- Tracing command audit pid 10 tid 100001 td 0xc8a3a620 sched_switch(c8a3a620,0,104,1a9,e4734bbc,...) at sched_switch+0x4e2/frame 0xe4734b84 mi_switch(104,0,c11e6431,200,c163975c,...) at mi_switch+0x1bf/frame 0xe4734bbc sleepq_switch(c8a3a620,0,c11e6431,276,c16397d0,...) at sleepq_switch+0x1c5/frame 0xe4734bf0 sleepq_wait(c16397d0,0,e4734c44,1,0,...) at sleepq_wait+0x48/frame 0xe4734c14 _cv_wait(c16397d0,c163975c,c1213466,18d,0,...) at _cv_wait+0x1fe/frame 0xe4734c58 audit_worker(0,e4734ce8,c11d90ec,3dc,0,...) at audit_worker+0x94/frame 0xe4734cac fork_exit(c0da1f00,0,e4734ce8) at fork_exit+0x7e/frame 0xe4734cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4734cd4 --- trap 0, eip = 0, esp = 0xe4734d20, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1632d90 sched_switch(c1632d90,0,104,1a9,c1c20c10,...) at sched_switch+0x4e2/frame 0xc1c20bd8 mi_switch(104,0,c11e6431,200,c163ae80,...) at mi_switch+0x1bf/frame 0xc1c20c10 sleepq_switch(c1632d90,0,c11e6431,276,c1632d90,...) at sleepq_switch+0x1c5/frame 0xc1c20c44 sleepq_wait(c1639a80,54,c12205ec,0,0,...) at sleepq_wait+0x48/frame 0xc1c20c68 _sleep(c1639a80,c163ae80,254,c12205ec,0,0,0,0,100) at _sleep+0x3a1/frame 0xc1c20cc4 vm_wait(c1632a44,0,54,c121f62a,fffff470,...) at vm_wait+0xfc/frame 0xc1c20cf0 swapper() at swapper+0x15/frame 0xc1c20d38 begin() at begin+0x22 Tracing command kernel pid 0 tid 100016 td 0xc8afb930 sched_switch(c8afb930,0,104,1a9,e4763bc0,...) at sched_switch+0x4e2/frame 0xe4763b88 mi_switch(104,0,c11e6431,200,c857be98,...) at mi_switch+0x1bf/frame 0xe4763bc0 sleepq_switch(c8afb930,0,c11e6431,276,c8afb930,...) at sleepq_switch+0x1c5/frame 0xe4763bf4 sleepq_wait(c857be80,0,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe4763c18 _sleep(c857be80,c857be98,0,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe4763c74 taskqueue_thread_loop(c1579ec4,e4763ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe4763cac fork_exit(c0bb2500,c1579ec4,e4763ce8) at fork_exit+0x7e/frame 0xe4763cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4763cd4 --- trap 0, eip = 0, esp = 0xe4763d20, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc8afb310 sched_switch(c8afb310,0,104,1a9,e4769bc0,...) at sched_switch+0x4e2/frame 0xe4769b88 mi_switch(104,0,c11e6431,200,c857bb98,...) at mi_switch+0x1bf/frame 0xe4769bc0 sleepq_switch(c8afb310,0,c11e6431,276,c8afb310,...) at sleepq_switch+0x1c5/frame 0xe4769bf4 sleepq_wait(c857bb80,0,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe4769c18 _sleep(c857bb80,c857bb98,0,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe4769c74 taskqueue_thread_loop(c1637110,e4769ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe4769cac fork_exit(c0bb2500,c1637110,e4769ce8) at fork_exit+0x7e/frame 0xe4769cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4769cd4 --- trap 0, eip = 0, esp = 0xe4769d20, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc8afac40 sched_switch(c8afac40,0,104,1a9,e476fbc0,...) at sched_switch+0x4e2/frame 0xe476fb88 mi_switch(104,0,c11e6431,200,c857ba98,...) at mi_switch+0x1bf/frame 0xe476fbc0 sleepq_switch(c8afac40,0,c11e6431,276,c8afac40,...) at sleepq_switch+0x1c5/frame 0xe476fbf4 sleepq_wait(c857ba80,0,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe476fc18 _sleep(c857ba80,c857ba98,0,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe476fc74 taskqueue_thread_loop(c1633298,e476fce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe476fcac fork_exit(c0bb2500,c1633298,e476fce8) at fork_exit+0x7e/frame 0xe476fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe476fcd4 --- trap 0, eip = 0, esp = 0xe476fd20, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc8afa930 sched_switch(c8afa930,0,104,1a9,e4772bd4,...) at sched_switch+0x4e2/frame 0xe4772b9c mi_switch(104,0,c11e6431,200,c157b97c,...) at mi_switch+0x1bf/frame 0xe4772bd4 sleepq_switch(c8afa930,0,c11e6431,276,c857ba00,...) at sleepq_switch+0x1c5/frame 0xe4772c08 sleepq_wait(c857ba00,0,c11e07ea,c11cafb6,0,...) at sleepq_wait+0x48/frame 0xe4772c2c msleep_spin_sbt(c857ba00,c857ba18,c11cafb6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe4772c74 taskqueue_thread_loop(c1629d78,e4772ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4772cac fork_exit(c0bb2500,c1629d78,e4772ce8) at fork_exit+0x7e/frame 0xe4772cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4772cd4 --- trap 0, eip = 0, esp = 0xe4772d20, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc8afa620 sched_switch(c8afa620,0,104,1a9,e4775bd4,...) at sched_switch+0x4e2/frame 0xe4775b9c mi_switch(104,0,c11e6431,200,c157b97c,...) at mi_switch+0x1bf/frame 0xe4775bd4 sleepq_switch(c8afa620,0,c11e6431,276,c857ba00,...) at sleepq_switch+0x1c5/frame 0xe4775c08 sleepq_wait(c857ba00,0,c11e07ea,c11cafb6,0,...) at sleepq_wait+0x48/frame 0xe4775c2c msleep_spin_sbt(c857ba00,c857ba18,c11cafb6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe4775c74 taskqueue_thread_loop(c1629d78,e4775ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4775cac fork_exit(c0bb2500,c1629d78,e4775ce8) at fork_exit+0x7e/frame 0xe4775cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4775cd4 --- trap 0, eip = 0, esp = 0xe4775d20, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc8afa310 sched_switch(c8afa310,0,104,1a9,e4778bd4,...) at sched_switch+0x4e2/frame 0xe4778b9c mi_switch(104,0,c11e6431,200,c157b97c,...) at mi_switch+0x1bf/frame 0xe4778bd4 sleepq_switch(c8afa310,0,c11e6431,276,c857ba00,...) at sleepq_switch+0x1c5/frame 0xe4778c08 sleepq_wait(c857ba00,0,c11e07ea,c11cafb6,0,...) at sleepq_wait+0x48/frame 0xe4778c2c msleep_spin_sbt(c857ba00,c857ba18,c11cafb6,0,0,...) at msleep_spin_sbt+0x241/frame 0xe4778c74 taskqueue_thread_loop(c1629d78,e4778ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4778cac fork_exit(c0bb2500,c1629d78,e4778ce8) at fork_exit+0x7e/frame 0xe4778cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4778cd4 --- trap 0, eip = 0, esp = 0xe4778d20, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc8be6930 sched_switch(c8be6930,0,104,1a9,e4784bc0,...) at sched_switch+0x4e2/frame 0xe4784b88 mi_switch(104,0,c11e6431,200,c857b818,...) at mi_switch+0x1bf/frame 0xe4784bc0 sleepq_switch(c8be6930,0,c11e6431,276,c8be6930,...) at sleepq_switch+0x1c5/frame 0xe4784bf4 sleepq_wait(c857b800,0,c11cafb6,0,0,...) at sleepq_wait+0x48/frame 0xe4784c18 _sleep(c857b800,c857b818,0,c11cafb6,0,...) at _sleep+0x3a1/frame 0xe4784c74 taskqueue_thread_loop(c1639804,e4784ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0x144/frame 0xe4784cac fork_exit(c0bb2500,c1639804,e4784ce8) at fork_exit+0x7e/frame 0xe4784cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4784cd4 --- trap 0, eip = 0, esp = 0xe4784d20, ebp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xc8c93930 sched_switch(c8c93930,0,104,1a9,ed96abd4,...) at sched_switch+0x4e2/frame 0xed96ab9c mi_switch(104,0,c11e6431,200,c157b934,...) at mi_switch+0x1bf/frame 0xed96abd4 sleepq_switch(c8c93930,0,c11e6431,276,c8cf3780,...) at sleepq_switch+0x1c5/frame 0xed96ac08 sleepq_wait(c8cf3780,0,c11e07ea,c11cafb6,0,...) at sleepq_wait+0x48/frame 0xed96ac2c msleep_spin_sbt(c8cf3780,c8cf3798,c11cafb6,0,0,...) at msleep_spin_sbt+0x241/frame 0xed96ac74 taskqueue_thread_loop(c1628214,ed96ace8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0xf3/frame 0xed96acac fork_exit(c0bb2500,c1628214,ed96ace8) at fork_exit+0x7e/frame 0xed96acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed96acd4 --- trap 0, eip = 0, esp = 0xed96ad20, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc8c93620 sched_switch(c8c93620,0,104,1a9,ed96db80,...) at sched_switch+0x4e2/frame 0xed96db48 mi_switch(104,0,c11e6431,200,101,...) at mi_switch+0x1bf/frame 0xed96db80 sleepq_switch(c8c93620,0,c11e6431,299,c8c93620,...) at sleepq_switch+0x1c5/frame 0xed96dbb4 sleepq_timedwait(c155ea4d,0,2,0,0,...) at sleepq_timedwait+0x48/frame 0xed96dbd8 _sleep(c155ea4d,0,0,c11cafb6,fffffc88,...) at _sleep+0x33d/frame 0xed96dc34 pause_sbt(c11cafb6,fffffc88,2,0,0,...) at pause_sbt+0x112/frame 0xed96dc70 deadlkres(0,ed96dce8,c11d90ec,3dc,0,...) at deadlkres+0x81/frame 0xed96dcac fork_exit(c0b055b0,0,ed96dce8) at fork_exit+0x7e/frame 0xed96dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed96dcd4 --- trap 0, eip = 0, esp = 0xed96dd20, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc8efd310 sched_switch(c8efd310,0,104,1a9,ed9748b8,...) at sched_switch+0x4e2/frame 0xed974880 mi_switch(104,0,c11e6431,200,c157d300,...) at mi_switch+0x1bf/frame 0xed9748b8 sleepq_switch(c8efd310,0,c11e6431,276,c157d380,...) at sleepq_switch+0x1c5/frame 0xed9748ec sleepq_wait(c157d380,0,ed974940,1,0,...) at sleepq_wait+0x48/frame 0xed974910 _cv_wait(c157d380,c157d300,c11e7a29,4b2,c121d821,...) at _cv_wait+0x1fe/frame 0xed974954 vmem_xalloc(c157d300,1000,0,0,0,...) at vmem_xalloc+0x1df/frame 0xed9749c8 vmem_alloc(c157d300,1000,2002,ed974a18,0,...) at vmem_alloc+0xc4/frame 0xed974a00 kmem_malloc(c157d300,1000,2) at kmem_malloc+0x3b/frame 0xed974a28 page_alloc(c1fa4b40,1000,ed974a6b,2,0,...) at page_alloc+0x28/frame 0xed974a3c keg_alloc_slab(102,4,c121d821,92a,c1fa3400,...) at keg_alloc_slab+0xd2/frame 0xed974a78 keg_fetch_slab(102,0,c121d821,975,c1fa4b40,...) at keg_fetch_slab+0x18f/frame 0xed974abc zone_fetch_slab(c1fa4b40,0,102,c0b65c20,c156be1c,...) at zone_fetch_slab+0x5d/frame 0xed974ae0 zone_import(c1fa4b40,ed974b40,1,102,0,...) at zone_import+0x37/frame 0xed974b28 zone_alloc_item(102,0,c121d838,c10c438f,1,...) at zone_alloc_item+0x39/frame 0xed974b58 uma_zalloc_arg(c1fa4b40,0,102,c8efd310,c8c743fc,...) at uma_zalloc_arg+0x658/frame 0xed974b98 malloc(47c,c13b937c,102,11bd,1,...) at malloc+0x1b1/frame 0xed974be4 xpt_run_allocq(c8c743fc,0,c10948ab,c40,c857b980,...) at xpt_run_allocq+0x1db/frame 0xed974c18 xpt_run_allocq_task(c8a30580,1,c11e6ceb,1ad,c857b990,...) at xpt_run_allocq_task+0x48/frame 0xed974c38 taskqueue_run_locked(c857b980,c857b998,0,c11cafb6,0,...) at taskqueue_run_locked+0x101/frame 0xed974c74 taskqueue_thread_loop(c1495b24,ed974ce8,c11d90ec,3dc,0,...) at taskqueue_thread_loop+0xa7/frame 0xed974cac fork_exit(c0bb2500,c1495b24,ed974ce8) at fork_exit+0x7e/frame 0xed974cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xed974cd4 --- trap 0, eip = 0, esp = 0xed974d20, ebp = 0 --- db:0:allt> call doadump Physical memory: 3431 MB Dumping 598 MB: 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) p vm_cnt $1 = {v_swtch = 0x0, v_trap = 0x0, v_syscall = 0x0, v_intr = 0x0, v_soft = 0x0, v_vm_faults = 0x0, v_io_faults = 0x0, v_cow_faults = 0x0, v_cow_optim = 0x0, v_zfod = 0x0, v_ozfod = 0x0, v_swapin = 0x0, v_swapout = 0x0, v_swappgsin = 0x0, v_swappgsout = 0x0, v_vnodein = 0x0, v_vnodeout = 0x0, v_vnodepgsin = 0x0, v_vnodepgsout = 0x0, v_intrans = 0x0, v_reactivated = 0x1b7bd4d, v_pdwakeups = 0xe4a, v_pdpages = 0x0, v_tcached = 0x0, v_dfree = 0x0, v_pfree = 0x0, v_tfree = 0x0, v_page_size = 0x1000, v_page_count = 0xd2d52, v_free_reserved = 0x496, v_free_target = 0x4806, v_free_min = 0x1572, v_free_count = 0x391, v_wire_count = 0x1fb7f, v_active_count = 0xb20b9, v_inactive_target = 0x6c09, v_inactive_count = 0x6, v_cache_count = 0xd7f, v_cache_min = 0x0, v_cache_max = 0x0, v_pageout_free_min = 0x22, v_interrupt_free_min = 0x2, v_free_severe = 0xd04, v_forks = 0x0, v_vforks = 0x0, v_rforks = 0x0, v_kthreads = 0x0, v_forkpages = 0x0, v_vforkpages = 0x0, v_rforkpages = 0x0, v_kthreadpages = 0x0} Current language: auto; currently minimal (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_malloc.c =================================================================== --- /usr/src/sys/kern/kern_malloc.c (revision 282344) +++ /usr/src/sys/kern/kern_malloc.c (working copy) @@ -672,6 +672,7 @@ static void kmem_reclaim(vmem_t *vm, int flags) { + uma_reclaim_needed = 1; pagedaemon_wakeup(); } Index: /usr/src/sys/vm/uma_core.c =================================================================== --- /usr/src/sys/vm/uma_core.c (revision 282344) +++ /usr/src/sys/vm/uma_core.c (working copy) @@ -3222,6 +3222,7 @@ uma_find_refcnt(uma_zone_t zone, void *item) } /* See uma.h */ +int pho2; void uma_reclaim(void) { @@ -3231,7 +3232,10 @@ uma_reclaim(void) sx_xlock(&uma_drain_lock); bucket_enable(); zone_foreach(zone_drain); - if (vm_page_count_min()) { + if (vm_page_count_min() || uma_reclaim_needed) { + if (uma_reclaim_needed) + pho2++; + uma_reclaim_needed = 0; cache_drain_safe(NULL); zone_foreach(zone_drain); } Index: /usr/src/sys/vm/vm_pageout.c =================================================================== --- /usr/src/sys/vm/vm_pageout.c (revision 282344) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -158,6 +158,7 @@ int vm_pages_needed; /* Event on which pageout da int vm_pageout_deficit; /* Estimated number of pages deficit */ int vm_pageout_pages_needed; /* flag saying that the pageout daemon needs pages */ int vm_pageout_wakeup_thresh; +int uma_reclaim_needed; #if !defined(NO_SWAPPING) static int vm_pageout_req_swapout; /* XXX */ @@ -1602,6 +1603,8 @@ vm_pageout_oom(int shortage) } } +int pho1, pho3; + static void vm_pageout_worker(void *arg) { @@ -1634,6 +1637,12 @@ vm_pageout_worker(void *arg) if (vm_pages_needed && !vm_page_count_min()) { if (!vm_paging_needed()) vm_pages_needed = 0; + if (uma_reclaim_needed) { + pho3++; + mtx_unlock(&vm_page_queue_free_mtx); + uma_reclaim(); + continue; + } wakeup(&vm_cnt.v_free_count); } if (vm_pages_needed) { @@ -1752,8 +1761,10 @@ void pagedaemon_wakeup(void) { - if (!vm_pages_needed && curthread->td_proc != pageproc) { + if ((!vm_pages_needed || uma_reclaim_needed) && + curthread->td_proc != pageproc) { vm_pages_needed = 1; + pho1++; wakeup(&vm_pages_needed); } } Index: /usr/src/sys/vm/vm_pageout.h =================================================================== --- /usr/src/sys/vm/vm_pageout.h (revision 282344) +++ /usr/src/sys/vm/vm_pageout.h (working copy) @@ -76,6 +76,7 @@ extern int vm_pages_needed; /* should be some "eve extern int vm_pageout_pages_needed; extern int vm_pageout_deficit; extern int vm_pageout_page_count; +extern int uma_reclaim_needed; /* * Swap out requests ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 256, 0, 196, 14, 3149, 0, 0 UMA Zones: 480, 0, 196, 4, 3149, 0, 0 UMA Slabs: 88, 0, 12092, 13, 115366, 0, 0 UMA RCntSlabs: 96, 0, 14737, 23, 15447, 0, 0 UMA Hash: 128, 0, 3, 28, 12, 0, 0 4 Bucket: 16, 0, 1139, 1861, 2380221, 0, 0 6 Bucket: 24, 0, 19, 1317, 103721, 0, 0 8 Bucket: 32, 0, 45, 2080, 927723, 0, 0 12 Bucket: 48, 0, 100, 979, 4193, 0, 0 16 Bucket: 64, 0, 147, 1465, 59706, 8, 0 32 Bucket: 128, 0, 1497, 766, 326501,19730, 0 64 Bucket: 256, 0, 209, 661, 122489, 0, 0 128 Bucket: 512, 0, 2091, 253, 1341504, 0, 0 256 Bucket: 1024, 0, 2216, 1228,10217827,225973, 0 vmem btag: 28, 0, 91124, 15983, 2063829, 748, 0 VM OBJECT: 156, 0, 4435, 113240,133391577, 0, 0 RADIX NODE: 44, 854399, 124265, 10506,121892706, 0, 0 MAP: 144, 0, 3, 78, 3, 0, 0 KMAP ENTRY: 76, 0, 5, 99, 5, 0, 0 MAP ENTRY: 76, 0, 1438, 13122,82755118, 0, 0 VMSPACE: 240, 0, 78, 2082, 3348065, 0, 0 fakepg: 68, 0, 0, 522, 760, 0, 0 mt_zone: 2060, 0, 372, 1, 377, 0, 0 16: 16, 0, 24, 476, 199256, 0, 0 16: 16, 0, 32, 1718,13687331, 0, 0 16: 16, 0, 70, 1680, 344247, 0, 0 16: 16, 0, 2631, 1119, 4279449, 0, 0 16: 16, 0, 38, 1212, 2817903, 0, 0 16: 16, 0, 94, 1656,14717624, 0, 0 16: 16, 0, 139, 1611,32164496, 0, 0 16: 16, 0, 50, 1450, 298299, 0, 0 32: 32, 0, 927, 198, 3259, 0, 0 32: 32, 0, 37, 338, 108, 0, 0 32: 32, 0, 50, 1075, 127095, 0, 0 32: 32, 0, 1247, 11753,149934674, 0, 0 32: 32, 0, 65, 1310, 5762466, 0, 0 32: 32, 0, 66, 1434, 8787913, 0, 0 32: 32, 0, 810, 68190,110018740, 0, 0 32: 32, 0, 894, 17481,18803902, 0, 0 64: 64, 0, 1748, 794, 18995, 0, 0 64: 64, 0, 33, 2075, 8642140, 0, 0 64: 64, 0, 258, 121696,225720838, 0, 0 64: 64, 0, 1458, 16770,245043109, 0, 0 64: 64, 0, 869, 22939,27884205, 0, 0 64: 64, 0, 6421, 523, 6690, 0, 0 64: 64, 0, 564, 9418,525065972, 0, 0 64: 64, 0, 120, 1554, 825545, 0, 0 128: 128, 0, 13, 49, 1063, 0, 0 128: 128, 0, 51, 17061,22009785, 0, 0 128: 128, 0, 127, 524, 80003, 0, 0 128: 128, 0, 214, 1026,92488544, 0, 0 128: 128, 0, 55, 503, 734220, 0, 0 128: 128, 0, 25587, 10249, 5726140, 0, 0 128: 128, 0, 720, 63419,51015621, 0, 0 128: 128, 0, 121, 49169,49241273, 0, 0 256: 256, 0, 6, 24, 6, 0, 0 256: 256, 0, 42, 128763,129023048, 0, 0 256: 256, 0, 90227, 253, 191538, 0, 0 256: 256, 0, 19257, 20043,37718693, 0, 0 256: 256, 0, 3064, 3506, 4882560, 0, 0 256: 256, 0, 2964, 246, 2464863, 0, 0 256: 256, 0, 514, 416, 1284217, 0, 0 256: 256, 0, 0, 255, 37427, 0, 0 512: 512, 0, 6, 10, 8, 0, 0 512: 512, 0, 43, 189, 7277910, 0, 0 512: 512, 0, 25, 183, 800049, 0, 0 512: 512, 0, 14, 90, 541017, 0, 0 512: 512, 0, 0, 176, 6069331, 0, 0 512: 512, 0, 1, 7, 9, 0, 0 512: 512, 0, 35, 261, 9737291, 0, 0 512: 512, 0, 6, 122, 947210, 0, 0 1024: 1024, 0, 3, 5, 1640, 0, 0 1024: 1024, 0, 2, 54, 130, 0, 0 1024: 1024, 0, 1, 71, 4466, 0, 0 1024: 1024, 0, 6, 54, 60092, 0, 0 1024: 1024, 0, 2, 94,638288982, 0, 0 1024: 1024, 0, 3, 3789, 3547122, 0, 0 1024: 1024, 0, 16, 16, 3701, 0, 0 1024: 1024, 0, 4, 28, 34695, 0, 0 2048: 2048, 0, 2, 2, 2, 0, 0 2048: 2048, 0, 1546, 66, 8382928, 2, 0 2048: 2048, 0, 20, 26, 454, 0, 0 2048: 2048, 0, 376, 30,45055544, 0, 0 2048: 2048, 0, 3, 33,89067197, 0, 0 2048: 2048, 0, 0, 6, 7, 0, 0 2048: 2048, 0, 9, 11, 3701, 0, 0 2048: 2048, 0, 3, 875, 204512, 0, 0 4096: 4096, 0, 6, 0, 6, 0, 0 4096: 4096, 0, 1, 0, 1, 0, 0 4096: 4096, 0, 3, 8, 395, 0, 0 4096: 4096, 0, 14, 9, 70795, 0, 0 4096: 4096, 0, 0, 0, 500107, 0, 0 4096: 4096, 0, 117, 969, 3348187, 0, 0 4096: 4096, 0, 26, 6, 9852, 0, 0 4096: 4096, 0, 2, 3, 133, 0, 0 8192: 8192, 0, 0, 0, 0, 0, 0 8192: 8192, 0, 2, 0, 2, 0, 0 8192: 8192, 0, 0, 0, 4, 0, 0 8192: 8192, 0, 10, 19,44479840, 0, 0 8192: 8192, 0, 0, 0, 0, 0, 0 8192: 8192, 0, 1, 1, 3, 0, 0 8192: 8192, 0, 1, 6, 7435, 0, 0 8192: 8192, 0, 4, 6, 457734, 0, 0 16384: 16384, 0, 1, 0, 1, 0, 0 16384: 16384, 0, 0, 2, 132, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 2, 6, 302881, 0, 0 16384: 16384, 0, 2, 1, 762, 0, 0 16384: 16384, 0, 1, 6269, 6228, 0, 0 16384: 16384, 0, 6, 5, 3788, 0, 0 16384: 16384, 0, 1, 0, 1, 0, 0 32768: 32768, 0, 1, 0, 1, 0, 0 32768: 32768, 0, 0, 40, 53543, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 32768: 32768, 0, 2, 7, 771, 0, 0 32768: 32768, 0, 0, 7, 122969, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 32768: 32768, 0, 1, 7, 4952, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 5, 6, 35134, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 1, 0, 342, 0, 0 65536: 65536, 0, 0, 4, 359, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 65536: 65536, 0, 10, 6, 3755, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 SLEEPQUEUE: 48, 0, 6386, 496, 6401, 0, 0 64 pcpu: 8, 0, 1449, 471, 1745, 0, 0 ptr pcpu: 4, 0, 0, 0, 0, 0, 0 Files: 56, 0, 431, 99040,199570911, 0, 0 filedesc0: 884, 0, 95, 1125, 3348154, 0, 0 rl_entry: 28, 0, 816, 900, 816, 0, 0 TURNSTILE: 72, 0, 6386, 341, 6401, 0, 0 umtx pi: 52, 0, 0, 308, 1, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 832, 0, 95, 2853, 3348154, 0, 0 THREAD: 784, 0, 3042, 3343, 3802302, 0, 0 cpuset: 40, 0, 112, 788, 35023, 0, 0 audit_record: 1112, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 168765, 257, 2803, 4159014, 0, 0 mbuf: 256, 168765, 1, 17212,70401867, 0, 0 mbuf_cluster: 2048, 26368, 3052, 54, 878247, 0, 0 mbuf_jumbo_page: 4096, 13184, 0, 13184,10837426, 0,24208635 mbuf_jumbo_9k: 9216, 11718, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 8788, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 164, 0, 170, 7174,149223866, 0, 0 ttyinq: 152, 0, 735, 227,19331115, 0, 0 ttyoutq: 256, 0, 387, 168,10309915, 0, 0 ata_request: 220, 0, 0, 306, 14713, 50, 0 vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 VNODE: 360, 0, 680, 71095,167642247, 0, 0 VNODEPOLL: 60, 0, 0, 1056, 334, 0, 0 BUF TRIE: 44, 0, 145, 5679,33516350, 0, 0 NAMEI: 1024, 0, 17, 595,628040744, 0, 0 S VFS Cache: 72, 0, 599, 60231,46044519, 0, 0 STS VFS Cache: 92, 0, 0, 0, 516455, 0, 0 L VFS Cache: 292, 0, 0, 2119, 661017, 0, 0 LTS VFS Cache: 312, 0, 0, 0, 9326, 0, 0 DIRHASH: 1024, 0, 394, 1326, 2051855, 0, 0 NCLNODE: 360, 0, 0, 0, 263141, 0, 0 pipe: 408, 0, 3, 393, 4137083, 0, 0 Mountpoints: 748, 0, 8, 87, 58913, 0, 0 procdesc: 76, 0, 0, 0, 21, 0, 0 ksiginfo: 80, 0, 2953, 847, 1571694, 107, 0 itimer: 232, 0, 0, 0, 0, 0, 0 KNOTE: 72, 0, 0, 3025, 8387311, 0, 0 socket: 456, 109808, 56, 272, 75617, 0, 0 unpcb: 172, 109825, 17, 259, 4176, 0, 0 ipq: 32, 875, 0, 0, 0, 0, 0 udp_inpcb: 256, 109815, 20, 190, 43066, 0, 0 udpcb: 16, 110000, 20, 1230, 43066, 0, 0 tcp_inpcb: 256, 109815, 15, 1515, 28323, 0, 0 tcpcb: 776, 109810, 15, 275, 28323, 0, 0 tcptw: 60, 21978, 0, 2310, 2233, 0, 0 syncache: 128, 15376, 0, 403, 8085, 0, 0 hostcache: 80, 15400, 0, 250, 8, 0, 0 sackhole: 20, 0, 0, 1200, 12527, 0, 0 sctp_ep: 984, 109808, 0, 0, 0, 0, 0 sctp_asoc: 1660, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80160, 0, 0, 3, 0, 0 sctp_raddr: 536, 80003, 0, 0, 0, 0, 0 sctp_chunk: 96, 400037, 0, 0, 0, 0, 0 sctp_readq: 76, 400036, 0, 0, 0, 0, 0 sctp_stream_msg_out: 68, 400026, 0, 0, 0, 0, 0 sctp_asconf: 24, 400132, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400132, 0, 0, 0, 0, 0 udplite_inpcb: 256, 109815, 0, 0, 0, 0, 0 ripcb: 256, 109815, 2, 253, 45, 0, 0 rtentry: 108, 0, 13, 98, 13, 0, 0 selfd: 28, 0, 229, 1344,12252448, 0, 0 SWAPMETA: 276, 431788, 2633, 81451, 403907, 0, 0 FFS inode: 120, 0, 610, 73178,159469489, 0, 0 FFS1 dinode: 128, 0, 0, 0, 2719855, 0, 0 FFS2 dinode: 256, 0, 610, 71570,156701782, 0, 0 Type InUse MemUse HighUse Requests Size(s) scsi_cd 0 0K - 8 16 hdaa 9 56K - 9 256,512,1024,2048,16384,32768 hdac 2 1K - 2 64,512 hdacc 2 1K - 2 16 feeder 21 1K - 27 16,64 mixer 4 16K - 4 4096 UART 3 2K - 3 16,512,1024 USB 36 15K - 36 16,64,256,2048 USBdev 30 3K - 30 32,128,256 ddb_capture 1 64K - 1 65536 acpiintr 1 1K - 1 32 acpica 2578 135K - 44524 16,32,64,128,256,512,1024 acpitask 1 4K - 1 4096 acpisem 32 4K - 32 64,128 vtbuf 24 1944K - 46 2048 vt 11 3K - 11 256 acpidev 31 1K - 31 32 DEVFS2 156 10K - 1872 16,32,64 DEVFS3 181 23K - 2352496 128 DEVFS1 158 40K - 1279231 256 DEVFS_RULE 55 13K - 55 32,256 DEVFS 32 1K - 182043 16,64 DEVFSP 1 1K - 100264 32 NFSD string 1 1K - 1 16 NFSD usrgroup 88 6K - 89 64 NFS fh 0 0K - 563356 32 NFSCL diroffdiroff 0 0K - 403 512 newnfsmnt 0 0K - 652 512 pfs_nodes 21 3K - 21 128 pfs_vncache 1 1K - 3200491 32 GEOM 1102 80K - 2951985 16,32,64,128,256,512,1024,2048,4096,8192 raid_data 0 0K - 627990 16,128 isadev 8 1K - 8 64 isofs_mount 0 0K - 1 128 isofs_node 0 0K - 5179 128 CAM SIM 6 1K - 6 128 CAM XPT 29 3K - 655 16,32,128,256,512,1024,2048,65536 CAM DEV 8 16K - 28 2048 CAM CCB 0 0K - 8381113 2048 CAM path 10 1K - 89 16 CAM periph 6 1K - 26 16,32,64,128 cdev 4 1K - 4 128 filedesc 10 69K - 80916 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 sigio 1 1K - 42 32 filecaps 0 0K - 150 16,64 kdtrace 3139 767K - 7150458 64,256 kenv 98 8K - 118 16,32,64,128,4096 kqueue 0 0K - 9092982 16,64,128,256,1024,2048,4096,16384 proc-args 56 3K - 1167297 16,32,64,128,256 hhook 9 2K - 9 128 ithread 83 7K - 83 16,64,128 prison 0 0K - 1 16 KTRACE 100 13K - 100 128 linker 175 622K - 384 16,32,64,256,512,1024,2048,4096,8192,16384,32768,65536 lockf 38 3K - 359381756 32,64,128,256 loginclass 3 1K - 3 64 cache 1 1K - 1 16 devbuf 2515 2872K - 283992 16,32,64,128,256,512,1024,2048,4096,8192,32768,65536 temp 31 227K - 743627507 16,32,64,128,256,512,1024,2048,16384,32768,65536 ip6opt 4 1K - 7 32,256 ip6ndp 4 1K - 5 64,128 pci_link 16 2K - 16 64,128 module 532 34K - 539 64,128 mtx_pool 2 8K - 2 4096 osd 2 1K - 2 16,32 pmchooks 1 1K - 1 64 pgrp 36 3K - 458 64 session 32 4K - 292 128 proc 2 8K - 2 4096 subproc 3044 1118K - 3351103 256,4096 cred 19206 4802K - 425450 256 plimit 16 4K - 295 256 uidinfo 4 2K - 322 64,1024 sysctl 0 0K - 1066968 16,32 sysctloid 3717 118K - 3810 16,32,64,128 sysctltmp 0 0K - 61695 16,32,64,256,1024 kbdmux 6 22K - 6 16,256,1024,2048,16384 tidhash 1 8K - 1 8192 callout 5 1308K - 5 umtx 12770 1198K - 12800 64,128 p1003.1b 1 1K - 1 16 SWAP 2 2189K - 2 64 bus 1311 57K - 5543845 16,32,64,128,256,1024 bus-sc 78 128K - 3372 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 devstat 28 57K - 28 16,4096 eventhandler 89 5K - 91 32,64,128 kobj 370 740K - 628652 2048 Per-cpu 1 1K - 1 16 rman 239 15K - 587 16,32,64 sbuf 0 0K - 93717074 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 stack 0 0K - 2 128 taskqueue 17 2K - 17 16,128 terminal 11 3K - 11 256 Unitno 45 2K - 13584146 16,64 vmem 4 593K - 370 64,128,256,512,1024,2048,4096,8192,16384,32768,65536 Witness 1539 3238K - 1539 2048,8192 ioctlops 0 0K - 1409 256,512,1024,2048 select 191 12K - 191 64 iov 0 0K - 518643277 16,64,128,256,512 msg 4 25K - 4 1024,4096,16384 sem 4 101K - 4 1024,4096 shm 1 16K - 1 16384 tty 39 20K - 1243928 512,2048 pts 26 4K - 1243913 128 mbuf_tag 0 0K - 11 32 shmfd 1 4K - 1 4096 soname 7 1K - 74030799 16,32,128 pcb 35 332K - 61462 16,32,64,512,1024,4096,65536 acl 0 0K - 1000146 512,4096 biobuf 2 4K - 99853 512,1024,2048 vfscache 1 512K - 1 cl_savebuf 0 0K - 4078982 32,64 export_host 10 2K - 270630 128 vfs_hash 1 256K - 1 vnodes 1 1K - 520 16,128 mount 19269 2400K - 26030736 16,32,64,128,256 md_disk 1 2K - 191 16,2048 vnodemarker 1 1K - 6032905 512 fadvise 0 0K - 1 64 BPF 2 1K - 2 64 ifnet 3 3K - 3 64,1024 ifaddr 43 8K - 43 16,32,64,128,256,1024,2048 ether_multi 47 2K - 47 16,32,64 clone 6 1K - 6 128 lltable 8 2K - 10 256 routetbl 90242 22557K - 180451 16,32,64,128,256 igmp 2 1K - 2 128 in_mfilter 1 1K - 1 512 in_multi 2 1K - 2 128 ip_moptions 2 1K - 2 32,128 sctp_a_it 0 0K - 2 16 sctp_vrf 1 1K - 1 64 sctp_ifa 4 1K - 4 128 sctp_ifn 2 1K - 2 128 sctp_iter 0 0K - 2 256 hostcache 1 16K - 1 16384 syncache 1 64K - 1 65536 in6_mfilter 3 2K - 3 512 in6_multi 27 2K - 27 16,128 ip6_moptions 6 1K - 6 32,128 mld 2 1K - 2 64 rpc 49 28K - 542827 16,32,64,256,512,1024,8192 audit_evclass 187 3K - 231 16 pagedep 10 321K - 21521210 128,65536 inodedep 24 1285K - 129057373 256 bmsafemap 6 21K - 89528462 128,4096 newblk 7 2561K - 48332830 128 indirdep 0 0K - 356773 64,16384,32768 freefrag 0 0K - 9689563 64 freeblks 0 0K - 11773538 128 freefile 0 0K - 88810306 32 diradd 16 1K - 108043205 64 mkdir 1 1K - 24194920 64 dirrem 11 1K - 107734858 64 newdirblk 1 1K - 12442693 32 freework 5 1K - 13583151 64,128 freedep 0 0K - 48757 32 jaddref 0 0K - 8337216 64 jremref 0 0K - 8336796 64 jnewblk 0 0K - 8530891 64 jfreefrag 0 0K - 1499388 64 jseg 0 0K - 71163 128 jsegdep 0 0K - 26704291 32 sbdep 0 0K - 188511 32 savedino 0 0K - 37253359 256 jblocks 2 5K - 11 128,256,512,1024,2048,4096 softdep 5 3K - 34842 512 ufs_dirhash 61 17K - 1292225 16,32,64,128,256,512,1024,2048,4096,8192 ufs_extattr 0 0K - 2 128 ufs_quota 1 256K - 1 ufs_mount 19 76K - 105569 256,2048,4096,8192,32768,65536 vm_pgdata 2 257K - 2 64 UMAHash 4 17K - 38 256,512,1024,2048,4096,8192,16384 md_sectors 0 0K - 123 4096 atkbddev 2 1K - 2 32 CAM queue 14 3K - 76 16,256 CAM dev queue 6 1K - 6 64 memdesc 1 4K - 1 4096 apmdev 2 1K - 244 64 madt_table 0 0K - 1 2048 io_apic 1 2K - 1 2048 entropy 2 1K - 2378 16,4096 MCA 6 1K - 6 128 msi 2 1K - 2 64 nexusdev 4 1K - 4 16 linux 15 1K - 15 32,64 tmpfs mount 0 0K - 1474 128 tmpfs name 0 0K - 179131 16,32 nullfs_hash 1 256K - 1 nullfs_node 0 0K - 1499844 32 nullfs_mount 0 0K - 16 16 fdesc_mount 0 0K - 14862 16 $ perl /home/pho/stress2/tools/calc_mem_use.pl < /tmp/vmstat.log Zones: 66703K ZoneFree: 359141K Malloc: 54212K Total: 480057K