GENERIC HEAD from 2016-03-09 11:45:48 UTC, r296564M, vmcore.917 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1 r296564M: Wed Mar 9 17:31:00 CET 2016 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3524550656 (3361 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. eval: limits: not found /etc/rc: WARNING: failed to start ddb warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2097312 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 714291 free (1275 frags, 89127 blocks, 0.1% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 717723 free (14651 frags, 87884 blocks, 1.4% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 70550892 free (27340 frags, 8815444 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1996884 free (239412 frags, 219684 blocks, 2.4% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 7190572 free (178660 frags, 876489 blocks, 0.7% fragmentation) Mounting local file systems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/gcc48 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/llvm35/lib /usr/local/llvm36/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy:. uhub5: 10 ports with 10 removable, self powered re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 scopeid 0x1 nd6 options=23 media: Ethernet autoselect (none) status: no carrier Starting devd. add net default: gateway 192.168.1.1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Starting rtsold. re0: link state changed to UP Creating and/or trimming log files. Starting syslogd. Starting watchdogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 Starting casperd. lock order reversal: 1st 0xe48f0ba8 bufwait (bufwait) @ kern/vfs_bio.c:3488 2nd 0xc96d3a00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:281 stack backtrace: #0 0xc0c918f1 at witness_debugger+0x81 #1 0xc0c9181a at witness_checkorder+0xd6a #2 0xc0c39cbb at _sx_xlock+0x6b #3 0xc0f06503 at ufsdirhash_add+0x53 #4 0xc0f0b162 at ufs_direnter+0x682 #5 0xc0f1410b at ufs_mkdir+0x7eb #6 0xc11caeee at VOP_MKDIR_APV+0x21e #7 0xc0cffc2d at kern_mkdirat+0x22d #8 0xc0cff9f1 at sys_mkdir+0x31 #9 0xc1197f68 at syscall+0x388 #10 0xc1182cef at Xint0x80_syscall+0x2f Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xca19ccdc ufs (ufs) @ kern/vfs_lookup.c:529 2nd 0xe4901984 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:263 3rd 0xca19ca0c ufs (ufs) @ kern/vfs_subr.c:2498 stack backtrace: #0 0xc0c918f1 at witness_debugger+0x81 #1 0xc0c9181a at witness_checkorder+0xd6a #2 0xc0c06df1 at __lockmgr_args+0xd71 #3 0xc0f00cc8 at ffs_lock+0x98 #4 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #5 0xc0d02faa at _vn_lock+0xba #6 0xc0cf08d2 at vget+0x122 #7 0xc0ce2a41 at vfs_hash_get+0xd1 #8 0xc0efb5f4 at ffs_vgetf+0x44 #9 0xc0ef2bbb at softdep_sync_buf+0xb0b #10 0xc0f01a2a at ffs_syncvnode+0x2aa #11 0xc0ed585b at ffs_truncate+0x99b #12 0xc0f0b2f3 at ufs_direnter+0x813 #13 0xc0f14eec at ufs_makeinode+0x52c #14 0xc0f10cb0 at ufs_create+0x30 #15 0xc11c66ce at VOP_CREATE_APV+0x21e #16 0xc0d027a5 at vn_open_cred+0x2a5 #17 0xc0d024ed at vn_open+0x3d . Updating motd:. Mounting late file systems:. Starting ntpd. Starting powerd. Starting smartd. Mar 9 17:40:09 x4 smartd[619]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Mar 9 17:40:09 x4 smartd[619]: see the following Seagate web pages: Mar 9 17:40:09 x4 smartd[619]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Mar 9 17:40:09 x4 smartd[619]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc9a10b74 ufs (ufs) @ kern/vfs_mount.c:1222 2nd 0xc974de44 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1907 stack backtrace: #0 0xc0c918f1 at witness_debugger+0x81 #1 0xc0c9181a at witness_checkorder+0xd6a #2 0xc0c06df1 at __lockmgr_args+0xd71 #3 0xc0cde9a3 at vop_stdlock+0x53 #4 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #5 0xc0d02faa at _vn_lock+0xba #6 0xc0edf478 at softdep_flushworklist+0x68 #7 0xc0eff0be at ffs_sync+0x2ae #8 0xc0d0470f at vfs_write_suspend+0x1cf #9 0xc0d04945 at vfs_write_suspend_umnt+0x45 #10 0xc0efe7b7 at ffs_unmount+0xa7 #11 0xc0ce8972 at dounmount+0x5f2 #12 0xc0ce8275 at sys_unmount+0x305 #13 0xc1197f68 at syscall+0x388 #14 0xc1182cef at Xint0x80_syscall+0x2f ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 390383 files, 47763089 used, 70550892 free (27324 frags, 8815446 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp. Starting default moused. Starting inetd. Wed Mar 9 17:42 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Mar 9 17:44:38 x4 su: pho to root on /dev/pts/0 root Password: Mar 9 17:44:56 x4 login: ROOT LOGIN (root) ON ttyu0 Last login: Wed Mar 9 17:33:21 on ttyu0 FreeBSD 11.0-CURRENT (PHO) #1 r296564M: Wed Mar 9 17:31:00 CET 2016 Welcome to FreeBSD! Release Notes, Errata: https://www.FreeBSD.org/releases/ Security Advisories: https://www.FreeBSD.org/security/ FreeBSD Handbook: https://www.FreeBSD.org/handbook/ FreeBSD FAQ: https://www.FreeBSD.org/faq/ Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/ FreeBSD Forums: https://forums.FreeBSD.org/ Documents installed with the system are in the /usr/local/share/doc/freebsd/ directory, or can be installed later with: pkg install en-freebsd-doc For other languages, replace "en" with a language code like de or fr. Show the version of FreeBSD installed: freebsd-version ; uname -a Please include that output and any error messages when posting questions. Introduction to manual pages: man man FreeBSD directory layout: man hier Edit /etc/motd to change this login announcement. You have new mail. root@x4:~ # bash $ cd ~pho/stress2/misc $ ./schedfuzz.sh kern.sched_fuzz.sched_fuzz_scheduler: off -> 0.01%return(1) kern.sched_fuzz.sched_fuzz_sx_slock_delay: off -> 0.01%schedfuzz(82) kern.sched_fuzz.sched_fuzz_sx_xlock_delay: off -> 0.01%schedfuzz(82) kern.sched_fuzz.sched_fuzz_sx_slock_sleep: off -> 0.01%schedfuzz(82) kern.sched_fuzz.sched_fuzz_sx_xlock_sleep: off -> 0.01%schedfuzz(82) kern.sched_fuzz.sched_fuzz_mtx_spin_lock: off -> 0.01%schedfuzz(44) kern.sched_fuzz.sched_fuzz_mtx_spin_unlock: off -> 0.01%schedfuzz(44) kern.sched_fuzz.sched_fuzz_malloc: off -> 0.01%schedfuzz(62) kern.sched_fuzz.sched_fuzz_cv_wait: off -> 0.1%schedfuzz(100) kern.sched_fuzz.sched_fuzz_cv_timed_wait: off -> 0.1%schedfuzz(100) kern.sched_fuzz.sched_fuzz_cv_timed_wait_sig: off -> 0.1%schedfuzz(100) kern.sched_fuzz.sched_fuzz_mtx_lock: off -> 0.01%schedfuzz(22) kern.sched_fuzz.sched_fuzz_mtx_unlock: off -> 0.01%schedfuzz(22) kern.sched_fuzz.sched_fuzz_sleepq: off -> 0.01%schedfuzz(150) kern.sched_fuzz.prof_on: 0 -> 1 Sleeping on "Sched fuzz fail point" with the following non-sleepable locks held: exclusive sleep mutex pfs_node (pfs_node) r = 0 (0xc8e40a20) locked @ fs/pseudofs/pseudofs_internal.h:103 stack backtrace: #0 0xc0c918f1 at witness_debugger+0x81 #1 0xc0c92ae8 at witness_warn+0x338 #2 0xc0c3b60f at _sleep+0x6f #3 0xc0c3bee8 at pause_sbt_pri+0x108 #4 0xc0bee0b4 at fail_point_eval_nontrivial+0x2e4 #5 0xc0c39b0f at _sx_slock+0x16f #6 0xc0c16b2e at pfind+0x2e #7 0xc0b6e582 at pfs_readdir+0x152 #8 0xc11cba4e at VOP_READDIR_APV+0x21e #9 0xc0cfff9b at kern_getdirentries+0x21b #10 0xc0cffd52 at sys_getdirentries+0x42 #11 0xc1197f68 at syscall+0x388 #12 0xc1182cef at Xint0x80_syscall+0x2f Sleeping on "Sched fuzz fail point" with the following non-sleepable locks held: exclusive sleep mutex pfs_node (pfs_node) r = 0 (0xc8e40a20) locked @ fs/pseudofs/pseudofs_internal.h:103 stack backtrace: #0 0xc0c918f1 at witness_debugger+0x81 #1 0xc0c92ae8 at witness_warn+0x338 #2 0xc0c3b60f at _sleep+0x6f #3 0xc0c3bee8 at pause_sbt_pri+0x108 #4 0xc0bee0b4 at fail_point_eval_nontrivial+0x2e4 #5 0xc0c39b0f at _sx_slock+0x16f #6 0xc0c16b2e at pfind+0x2e #7 0xc0b6e582 at pfs_readdir+0x152 #8 0xc11cba4e at VOP_READDIR_APV+0x21e #9 0xc0cfff9b at kern_getdirentries+0x21b #10 0xc0cffd52 at sys_getdirentries+0x42 #11 0xc1197f68 at syscall+0x388 #12 0xc1182cef at Xint0x80_syscall+0x2f Sleeping thread (tid 100239, pid 1798) owns a non-sleepable lock KDB: stack backtrace of thread 100239: sched_switch(caece340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a658bc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3a658f4 sleepq_switch(caece340,0,c1583a69,29c,caece340,...) at sleepq_switch+0x127/frame 0xf3a65928 sleepq_timedwait(c19be8c1,0,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf3a6594c _sleep(c19be8c1,0,400,c1575203,b020c3d,...) at _sleep+0x362/frame 0xf3a659ac pause_sbt_pri(c1575203,400,b020c3d,0,0,...) at pause_sbt_pri+0x108/frame 0xf3a659e8 fail_point_eval_nontrivial(c1820938,f3a65a60,c157ab75,144,0,...) at fail_point_eval_nontrivial+0x2e4/frame 0xf3a65a40 _sx_slock(c1a93ef0,0,c157ab75,144,1000,...) at _sx_slock+0x16f/frame 0xf3a65a74 pfind(0,0,c1566c5f,67,0,...) at pfind+0x2e/frame 0xf3a65a90 pfs_readdir(f3a65bc0,c1770ce4,f3a65b10,200400,cba5e2d0,...) at pfs_readdir+0x152/frame 0xf3a65af4 VOP_READDIR_APV(c180f27c,f3a65bc0,c1591230,f9c,0,...) at VOP_READDIR_APV+0x21e/frame 0xf3a65b50 kern_getdirentries(caece340,5,2864d000,1000,f3a65c10,...) at kern_getdirentries+0x21b/frame 0xf3a65bec sys_getdirentries(caece340,f3a65ca8,c1574eec,4,c119f5d9,...) at sys_getdirentries+0x42/frame 0xf3a65c20 syscall(f3a65ce8) at syscall+0x388/frame 0xf3a65cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a65cdc --- syscall (196, FreeBSD ELF32, sys_getdirentries), eip = 0x28220cbb, esp = 0xbfbfe0c0, ebp = 0xbfbfe0f8 --- panic: sleeping thread cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c1581e68,0,c15557c8,20c,f3975468,...) at db_trace_self_wrapper+0x2a/frame 0xf3975438 kdb_backtrace(c176ef26,0,c1584bdc,f39754f4,0,...) at kdb_backtrace+0x2d/frame 0xf39754a0 vpanic(c1584bdc,f39754f4,f39754f4,f3975580,c0c85f5b,...) at vpanic+0x115/frame 0xf39754d4 panic(c1584bdc,1878f,706,d3,c19dd2c8,...) at panic+0x1b/frame 0xf39754e8 propagate_priority(ca3d6680,0,c15847f0,2e5,c96f1800,...) at propagate_priority+0x2db/frame 0xf3975580 turnstile_wait(c96f1800,caece340,0,202,c180f27c,...) at turnstile_wait+0x80b/frame 0xf3975640 __mtx_lock_sleep(c8e40a30,ca3d6680,0,c1566c5f,67,...) at __mtx_lock_sleep+0x727/frame 0xf3975788 __mtx_lock_flags(c8e40a30,0,c1566c5f,67,0,...) at __mtx_lock_flags+0x225/frame 0xf39757e8 pfs_lookup(f39758d0,c177058e,c0cde7a8,cb99c5d4,f39758c8,...) at pfs_lookup+0x1d6/frame 0xf3975854 VOP_CACHEDLOOKUP_APV(c180f27c,f39758d0,f3975b44,0,0,...) at VOP_CACHEDLOOKUP_APV+0x21e/frame 0xf39758b0 vfs_cache_lookup(f3975990,c177052d,c95e92ec,80000,f3975938,...) at vfs_cache_lookup+0xe9/frame 0xf39758f4 VOP_LOOKUP_APV(c180f27c,f3975990,f3975b44,211,c1a09740,...) at VOP_LOOKUP_APV+0x21e/frame 0xf3975950 lookup(f3975ae8,c158ed16,124,eb,cbe02400,...) at lookup+0x53d/frame 0xf39759b8 namei(f3975ae8,0,4000104,0,28669db8,...) at namei+0x5df/frame 0xf3975a48 kern_statat(ca3d6680,200,ffffff9c,28669db8,0,...) at kern_statat+0xae/frame 0xf3975b80 sys_fstatat(ca3d6680,f3975ca8,c1574eec,4,c119f5d9,...) at sys_fstatat+0x49/frame 0xf3975c20 syscall(f3975ce8) at syscall+0x388/frame 0xf3975cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3975cdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- KDB: enter: panic [ thread pid 1838 tid 100179 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho script 'pho' not found db> set $lines 0 db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x7c6100 curthread = 0xca3d6680: pid 1838 "ls" curpcb = 0xf3975d40 fpcurthread = none idlethread = 0xc8e48340: tid 100004 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x232b9100 curthread = 0xcbe3b340: pid 2025 "mount" curpcb = 0xf3a87d40 fpcurthread = none idlethread = 0xc8e48000: tid 100005 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x232bc100 curthread = 0xca18b680: pid 2038 "date" curpcb = 0xf39b2d40 fpcurthread = none idlethread = 0xc8e479c0: tid 100006 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x232bf100 curthread = 0xcbe70000: pid 2013 "mount" curpcb = 0xf3ad8d40 fpcurthread = 0xcbe70000: pid 2013 "mount" idlethread = 0xc8e47680: tid 100007 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db> show alllocks Process 2039 (date) thread 0xcbd6b340 (100257) shared lockmgr ufs (ufs) r = 0 (0xc9a0fcdc) locked @ kern/vfs_subr.c:2498 Process 2038 (date) thread 0xca18b680 (100197) shared sx vm map (user) (vm map (user)) r = 0 (0xca0fe31c) locked @ vm/vm_map.c:3985 Process 2037 (ls) thread 0xcbd6c000 (100238) shared sx vm map (user) (vm map (user)) r = 0 (0xca0fd40c) locked @ vm/vm_map.c:3985 Process 2036 (umount) thread 0xcaecf680 (100203) shared rw vm object (vm object) r = 0 (0xc96b01e0) locked @ vm/vm_map.c:1875 exclusive sx vm map (user) (vm map (user)) r = 0 (0xc96d98bc) locked @ vm/vm_map.c:1505 Process 2035 (ls) thread 0xcbdaf000 (100251) shared lockmgr ufs (ufs) r = 0 (0xc9a0fcdc) locked @ kern/vfs_subr.c:2498 Process 2034 (grep) thread 0xcbdaf340 (100250) shared rw vm object (vm object) r = 0 (0xc96ca6e0) locked @ vm/vm_map.c:1875 exclusive sx vm map (user) (vm map (user)) r = 0 (0xc97bc40c) locked @ vm/vm_map.c:1505 Process 2033 (grep) thread 0xca3d69c0 (100178) exclusive sx vm map (user) (vm map (user)) r = 0 (0xc97bcc7c) locked @ vm/vm_map.c:1458 Process 2031 (mount) thread 0xcbe3c340 (100275) shared lockmgr ufs (ufs) r = 0 (0xc9685e44) locked @ kern/vfs_subr.c:2498 Process 2030 (grep) thread 0xcbe9b000 (100278) exclusive sx vm map (user) (vm map (user)) r = 0 (0xca0fe7cc) locked @ vm/vm_map.c:1505 Process 2029 (umount) thread 0xcbe71680 (100265) shared rw vm object (vm object) r = 0 (0xc96b0a00) locked @ vm/vm_map.c:1875 exclusive sx vm map (user) (vm map (user)) r = 0 (0xc8e51c7c) locked @ vm/vm_map.c:1505 Process 2027 (mount) thread 0xcbe9b340 (100277) shared lockmgr ufs (ufs) r = 0 (0xc9a0fcdc) locked @ kern/vfs_subr.c:2498 Process 2026 (ls) thread 0xca18b9c0 (100194) exclusive sx vm map (user) (vm map (user)) r = 0 (0xc973d9ac) locked @ vm/vm_mmap.c:550 Process 2025 (mount) thread 0xcbe3b340 (100248) shared sx vm map (user) (vm map (user)) r = 0 (0xc97bcb8c) locked @ vm/vm_map.c:3985 Process 2024 (umount) thread 0xc973c340 (100090) shared sx vm map (user) (vm map (user)) r = 0 (0xc97bac7c) locked @ vm/vm_map.c:3985 Process 2023 (umount) thread 0xcbd6c340 (100237) shared sx vm map (user) (vm map (user)) r = 0 (0xca0fdb8c) locked @ vm/vm_map.c:3985 Process 2022 (grep) thread 0xcbd6c680 (100232) shared sx vm map (user) (vm map (user)) r = 0 (0xca0fe40c) locked @ vm/vm_map.c:3985 Process 2021 (grep) thread 0xcbe3c9c0 (100273) shared lockmgr ufs (ufs) r = 0 (0xc9b81a0c) locked @ kern/vfs_subr.c:2498 Process 2019 (mount) thread 0xcbe9b680 (100276) shared rw vm object (vm object) r = 0 (0xc95c5dc0) locked @ vm/vm_map.c:1875 exclusive sx vm map (user) (vm map (user)) r = 0 (0xc973f4fc) locked @ vm/vm_map.c:1505 Process 2018 (grep) thread 0xcbe70340 (100271) exclusive sx vm map (user) (vm map (user)) r = 0 (0xca0fd7cc) locked @ vm/vm_map.c:1458 Process 2016 (umount) thread 0xcbd6b680 (100241) shared sx vm map (user) (vm map (user)) r = 0 (0xc973f40c) locked @ vm/vm_map.c:3985 Process 2014 (grep) thread 0xcaed0000 (100193) shared rw vm object (vm object) r = 0 (0xc95c4c80) locked @ vm/vm_map.c:1875 exclusive sx vm map (user) (vm map (user)) r = 0 (0xca0fd22c) locked @ vm/vm_map.c:1505 Process 2013 (mount) thread 0xcbe70000 (100272) exclusive sleep mutex pmap (pmap) r = 0 (0xc97bb924) locked @ i386/i386/pmap.c:4482 exclusive rw pmap pv global (pmap pv global) r = 0 (0xc1a51f00) locked @ i386/i386/pmap.c:4481 Process 2001 (ls) thread 0xcbe3c680 (100274) shared sx vm map (user) (vm map (user)) r = 0 (0xc8e504fc) locked @ vm/vm_map.c:3985 Process 1838 (ls) thread 0xca3d6680 (100179) exclusive lockmgr pseudofs (pseudofs) r = 0 (0xcb99c5d4) locked @ kern/vfs_lookup.c:529 Process 1798 (ls) thread 0xcaece340 (100239) exclusive sleep mutex pfs_node (pfs_node) r = 0 (0xc8e40a20) locked @ fs/pseudofs/pseudofs_internal.h:103 shared sx allproc (allproc) r = 0 (0xc1a93ef0) locked @ fs/pseudofs/pseudofs_vnops.c:790 exclusive lockmgr pseudofs (pseudofs) r = 0 (0xcba5e304) locked @ kern/vfs_syscalls.c:3996 Process 1787 (ls) thread 0xcbe71000 (100267) exclusive lockmgr pseudofs (pseudofs) r = 0 (0xcb8a3304) locked @ kern/vfs_subr.c:2498 Process 1065 (sh) thread 0xcbd89680 (100216) exclusive sx proctree (proctree) r = 0 (0xc1a93eac) locked @ kern/kern_exit.c:1152 db> show lockedvnods Locked vnodes 0xc9685e10: tag ufs, type VREG usecount 11, writecount 0, refcount 12 mountedhere 0 flags (VI_ACTIVE) v_object 0xc96b0a00 ref 9 pages 1 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xc0c06d7f at __lockmgr_args+0xcff #1 0xc0f00cc8 at ffs_lock+0x98 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0cf08d2 at vget+0x122 #5 0xc0cd956d at cache_lookup+0x7fd #6 0xc0cdab35 at vfs_cache_lookup+0xb5 #7 0xc11c5fae at VOP_LOOKUP_APV+0x21e #8 0xc0ce463d at lookup+0x53d #9 0xc0ce3cdf at namei+0x5df #10 0xc0d026fd at vn_open_cred+0x1fd #11 0xc0d024ed at vn_open+0x3d #12 0xc0cfa971 at kern_openat+0x361 #13 0xc0cfac6a at sys_openat+0x3a #14 0xc1197f68 at syscall+0x388 #15 0xc1182cef at Xint0x80_syscall+0x2f ino 23742, on dev ada0s1a 0xc9a0fca8: tag ufs, type VREG usecount 8, writecount 0, refcount 9 mountedhere 0 flags (VI_ACTIVE) v_object 0xc96ca6e0 ref 3 pages 1 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 3) #0 0xc0c06d7f at __lockmgr_args+0xcff #1 0xc0f00cc8 at ffs_lock+0x98 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0cf08d2 at vget+0x122 #5 0xc0cd956d at cache_lookup+0x7fd #6 0xc0cdab35 at vfs_cache_lookup+0xb5 #7 0xc11c5fae at VOP_LOOKUP_APV+0x21e #8 0xc0ce463d at lookup+0x53d #9 0xc0ce3cdf at namei+0x5df #10 0xc0d026fd at vn_open_cred+0x1fd #11 0xc0d024ed at vn_open+0x3d #12 0xc0cfa971 at kern_openat+0x361 #13 0xc0cfac6a at sys_openat+0x3a #14 0xc1197f68 at syscall+0x388 #15 0xc1182cef at Xint0x80_syscall+0x2f ino 1460225, on dev ada0s1d 0xc9b819d8: tag ufs, type VREG usecount 12, writecount 0, refcount 13 mountedhere 0 flags (VI_ACTIVE) v_object 0xc95c4c80 ref 8 pages 17 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 1) #0 0xc0c06d7f at __lockmgr_args+0xcff #1 0xc0f00cc8 at ffs_lock+0x98 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0cf08d2 at vget+0x122 #5 0xc0cd956d at cache_lookup+0x7fd #6 0xc0cdab35 at vfs_cache_lookup+0xb5 #7 0xc11c5fae at VOP_LOOKUP_APV+0x21e #8 0xc0ce463d at lookup+0x53d #9 0xc0ce3cdf at namei+0x5df #10 0xc0d026fd at vn_open_cred+0x1fd #11 0xc0d024ed at vn_open+0x3d #12 0xc0cfa971 at kern_openat+0x361 #13 0xc0cfac6a at sys_openat+0x3a #14 0xc1197f68 at syscall+0x388 #15 0xc1182cef at Xint0x80_syscall+0x2f ino 283584, on dev ada0s1d 0xcb8a32d0: tag pseudofs, type VDIR usecount 14, writecount 0, refcount 18 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) lock type pseudofs: EXCL by thread 0xcbe71000 (pid 1787, ls, tid 100267) with exclusive waiters pending #0 0xc0c0710f at __lockmgr_args+0x108f #1 0xc0cde9a3 at vop_stdlock+0x53 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0cf08d2 at vget+0x122 #5 0xc0cd956d at cache_lookup+0x7fd #6 0xc0cdab35 at vfs_cache_lookup+0xb5 #7 0xc11c5fae at VOP_LOOKUP_APV+0x21e #8 0xc0ce463d at lookup+0x53d #9 0xc0ce3cdf at namei+0x5df #10 0xc0d026fd at vn_open_cred+0x1fd #11 0xc0d024ed at vn_open+0x3d #12 0xc0cfa971 at kern_openat+0x361 #13 0xc0cfa609 at sys_open+0x39 #14 0xc1197f68 at syscall+0x388 #15 0xc1182cef at Xint0x80_syscall+0x2f 0xcb99c5a0: tag pseudofs, type VDIR usecount 3, writecount 0, refcount 3 mountedhere 0 flags (VV_PROCDEP|VI_ACTIVE) lock type pseudofs: EXCL by thread 0xca3d6680 (pid 1838, ls, tid 100179) #0 0xc0c0710f at __lockmgr_args+0x108f #1 0xc0cde9a3 at vop_stdlock+0x53 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0ce41f1 at lookup+0xf1 #5 0xc0ce3cdf at namei+0x5df #6 0xc0cfca6e at kern_statat+0xae #7 0xc0cfcc39 at sys_fstatat+0x49 #8 0xc1197f68 at syscall+0x388 #9 0xc1182cef at Xint0x80_syscall+0x2f 0xcba5e2d0: tag pseudofs, type VDIR usecount 5, writecount 0, refcount 6 mountedhere 0 flags (VV_PROCDEP|VI_ACTIVE) lock type pseudofs: EXCL by thread 0xcaece340 (pid 1798, ls, tid 100239) with exclusive waiters pending #0 0xc0c0710f at __lockmgr_args+0x108f #1 0xc0cde9a3 at vop_stdlock+0x53 #2 0xc11cc7c2 at VOP_LOCK1_APV+0x202 #3 0xc0d02faa at _vn_lock+0xba #4 0xc0cfff0d at kern_getdirentries+0x18d #5 0xc0cffd52 at sys_getdirentries+0x42 #6 0xc1197f68 at syscall+0x388 #7 0xc1182cef at Xint0x80_syscall+0x2f db> show mount 0xc96a0000 /dev/ada0s1a on / (ufs) 0xc95eb5d8 devfs on /dev (devfs) 0xc95eb2ec /dev/ada0s1f on /home (ufs) 0xc95eabb0 /dev/ada0s1d on /usr (ufs) 0xc95ea8c4 /dev/ada0s1e on /var (ufs) 0xc969fbb0 procfs on /proc (procfs) 0xc969f8c4 /dev/label/tmp on /tmp (ufs) 0xc95e92ec procfs on /mnt9 (procfs) 0xcbe945d8 procfs on /mnt13 (procfs) 0xc95e9bb0 procfs on /mnt7 (procfs) 0xcbe3ebb0 procfs on /mnt8 (procfs) 0xcbe108c4 procfs on /mnt18 (procfs) 0xcbe942ec procfs on /mnt10 (procfs) 0xcbe94000 procfs on /mnt17 (procfs) 0xc95e9000 procfs on /mnt19 (procfs) 0xc95ea000 procfs on /mnt6 (procfs) More info: show mount db> ps pid ppid pgrp uid state wmesg wchan cmd 2039 2028 960 0 L+ *vm objec 0xc8cdef00 date 2038 2032 960 0 R+ CPU 2 date 2037 1064 960 0 R+ ls 2036 1068 960 0 R+ umount 2035 1053 960 0 L+ *vm objec 0xc8cdef00 ls 2034 1033 960 0 R+ grep 2033 1072 960 0 R+ grep 2032 1070 960 0 S+ piperd 0xcbdda000 sh 2031 1033 960 0 L+ *vm objec 0xc8cde400 mount 2030 1047 960 0 L+ *vm objec 0xc8cdef00 grep 2029 1052 960 0 R+ umount 2028 1044 960 0 S+ piperd 0xc97b7000 sh 2027 1072 960 0 L+ *vm objec 0xc8cdef00 mount 2026 1055 960 0 L+ *vm objec 0xc8cdef00 ls 2025 1047 960 0 R+ CPU 1 mount 2024 1039 960 0 R+ umount 2023 1037 960 0 R+ umount 2022 1065 960 0 R+ grep 2021 1049 960 0 L+ *vm objec 0xc952f680 grep 2019 1049 960 0 R+ mount 2018 1056 960 0 L+ *vm objec 0xc952f680 grep 2016 1041 960 0 R+ umount 2014 1054 960 0 R+ grep 2013 1035 960 0 RE+ CPU 3 mount 2001 1038 960 0 R+ ls 1838 1046 960 0 L+ *pfs_node 0xc96f1800 ls 1830 1050 960 0 D+ pseudofs 0xcb8a3304 ls 1829 1071 960 0 D+ pseudofs 0xcb8a3304 ls 1825 1066 960 0 D+ pseudofs 0xcb8a3304 ls 1798 1042 960 0 D+ Sched fu 0xc19be8c1 ls 1796 1034 960 0 D+ pseudofs 0xcb8a3304 ls 1793 1048 960 0 D+ pseudofs 0xcb8a3304 ls 1790 1040 960 0 D+ pseudofs 0xcb8a3304 ls 1789 1036 960 0 D+ pseudofs 0xcb8a3304 ls 1787 1073 960 0 D+ pseudofs 0xcba5e304 ls 1771 1069 960 0 D+ pseudofs 0xcb8a3304 ls 1073 978 960 0 S+ wait 0xcaeeb358 sh 1072 978 960 0 S+ wait 0xcaeeb6b0 sh 1071 978 960 0 S+ wait 0xcaeeba08 sh 1070 978 960 0 S+ piperd 0xc97b7330 sh 1069 978 960 0 S+ wait 0xcbd9d358 sh 1068 978 960 0 S+ wait 0xcbd9d6b0 sh 1066 978 960 0 S+ wait 0xcbd9da08 sh 1065 978 960 0 D+ allproc 0xc1a93ef0 sh 1064 978 960 0 S+ wait 0xcbd9e358 sh 1056 978 960 0 S+ wait 0xcbd9ea08 sh 1055 978 960 0 S+ wait 0xcbd9f000 sh 1054 978 960 0 S+ wait 0xcbd9f358 sh 1053 978 960 0 S+ wait 0xc96e9358 sh 1052 978 960 0 S+ wait 0xc9aa26b0 sh 1050 978 960 0 S+ wait 0xc9aa2358 sh 1049 978 960 0 S+ wait 0xc9aa2000 sh 1048 978 960 0 S+ wait 0xca2d66b0 sh 1047 978 960 0 S+ wait 0xca2d6358 sh 1046 978 960 0 S+ wait 0xca2d6000 sh 1044 978 960 0 S+ piperd 0xc9784660 sh 1042 978 960 0 S+ wait 0xc9378a08 sh 1041 978 960 0 S+ wait 0xcaee8000 sh 1040 978 960 0 S+ wait 0xcaee8358 sh 1039 978 960 0 S+ wait 0xcaee86b0 sh 1038 978 960 0 S+ wait 0xcaee8a08 sh 1037 978 960 0 S+ wait 0xcaee9000 sh 1036 978 960 0 S+ wait 0xcaee9358 sh 1035 978 960 0 S+ wait 0xcaee96b0 sh 1034 978 960 0 S+ wait 0xca2d56b0 sh 1033 978 960 0 S+ wait 0xc9a98000 sh 978 960 960 0 S+ pause 0xcaeea714 sh 960 954 960 0 S+ wait 0xcaee9a08 sh 954 952 954 0 S+ wait 0xcaeea000 bash 952 809 952 0 S+ pause 0xcaeea3bc csh 932 931 932 0 S+ ttyin 0xcbd4fc70 csh 931 924 931 1001 S+ wait 0xca2d5000 su 924 923 924 1001 Ss+ wait 0xc97286b0 bash 923 921 921 1001 S select 0xc96ad164 sshd 921 658 921 0 Ss select 0xc95fb6e4 sshd 809 1 809 0 Ss+ wait 0xc96e7000 login 808 1 808 0 Ss+ ttyin 0xc90f7270 getty 807 1 807 0 Ss+ ttyin 0xc90f7470 getty 806 1 806 0 Ss+ ttyin 0xc90f7670 getty 805 1 805 0 Ss+ ttyin 0xc90f7870 getty 804 1 804 0 Ss+ ttyin 0xc90f7a70 getty 803 1 803 0 Ss+ ttyin 0xc90f7c70 getty 802 1 802 0 Ss+ ttyin 0xc90f7e70 getty 801 1 801 0 Ss+ ttyin 0xc90f8070 getty 782 1 782 0 Ss select 0xc96adb24 inetd 752 1 752 0 Ss select 0xc9111124 moused 669 1 669 0 Ss nanslp 0xc19bf9c3 cron 664 1 664 25 Ss pause 0xc8e453bc sendmail 661 1 661 0 Ss select 0xc96ad724 sendmail 658 1 658 0 Ss select 0xc9111164 sshd 621 1 620 0 S nanslp 0xc19bf9c0 smartd 602 1 602 0 Ss select 0xc96adba4 powerd 599 1 599 0 Ss (threaded) ntpd 100152 S select 0xc96adaa4 ntpd 545 538 538 0 S (threaded) nfsd 100119 S rpcsvc 0xca18f390 nfsd: master 100120 S rpcsvc 0xca17a190 nfsd: service 100121 S rpcsvc 0xca17e290 nfsd: service 100122 S rpcsvc 0xca17a090 nfsd: service 100123 S rpcsvc 0xca17e190 nfsd: service 100124 S rpcsvc 0xca178e90 nfsd: service 100125 S rpcsvc 0xca178d90 nfsd: service 100126 S rpcsvc 0xca178c90 nfsd: service 100127 S rpcsvc 0xca17e090 nfsd: service 100128 S rpcsvc 0xca178b90 nfsd: service 100129 S rpcsvc 0xca17de90 nfsd: service 100130 S rpcsvc 0xca178a90 nfsd: service 100131 S rpcsvc 0xca17dd90 nfsd: service 100132 S rpcsvc 0xca178990 nfsd: service 100133 S rpcsvc 0xca178890 nfsd: service 100134 S rpcsvc 0xca17dc90 nfsd: service 100135 S rpcsvc 0xca178790 nfsd: service 100136 S rpcsvc 0xca178690 nfsd: service 100137 S rpcsvc 0xca17db90 nfsd: service 100138 S rpcsvc 0xca17da90 nfsd: service 100139 S rpcsvc 0xca178590 nfsd: service 100140 S rpcsvc 0xca17d990 nfsd: service 100141 S rpcsvc 0xca178490 nfsd: service 100142 S rpcsvc 0xca178390 nfsd: service 100143 S rpcsvc 0xca17d890 nfsd: service 100144 S rpcsvc 0xca17d790 nfsd: service 100145 S rpcsvc 0xca17d690 nfsd: service 100146 S rpcsvc 0xca17d590 nfsd: service 100147 S rpcsvc 0xca17d490 nfsd: service 100148 S rpcsvc 0xca178290 nfsd: service 100149 S rpcsvc 0xca17d390 nfsd: service 100150 S rpcsvc 0xca17d290 nfsd: service 538 1 538 0 Ss select 0xc96ae024 nfsd 532 1 532 0 Ss select 0xc9538c64 mountd 525 521 521 0 S select 0xc96ae664 nfsuserd 524 521 521 0 S select 0xc95377a4 nfsuserd 523 521 521 0 S select 0xc95fc524 nfsuserd 522 521 521 0 S select 0xc9537724 nfsuserd 521 1 521 0 Ss pause 0xc9729064 nfsuserd 504 1 504 0 Ss select 0xc9537864 casperd 503 1 503 0 Ss select 0xc95378a4 casperd 489 1 489 0 Ss select 0xc9537824 rpcbind 472 1 472 0 Ss nanslp 0xc19bf9c2 watchdogd 468 1 468 0 Ss select 0xc95378e4 syslogd 359 1 359 0 Ss select 0xc95386a4 rtsold 318 1 318 0 Ss select 0xc95fb7e4 devd 23 0 0 0 DL syncer 0xc1a39d10 [syncer] 22 0 0 0 DL vlruwt 0xc9376358 [vnlru] 21 0 0 0 DL (threaded) [bufdaemon] 100085 D psleep 0xc1a38b04 [bufdaemon] 100098 D sdflush 0xc9a1c284 [/home worker] 100100 D sdflush 0xc9a1be84 [/usr worker] 100101 D sdflush 0xc9afec84 [/var worker] 100167 D sdflush 0xc9afea84 [/tmp worker] 20 0 0 0 DL - 0xc1a39748 [bufspacedaemon] 19 0 0 0 DL pgzero 0xc1a3f2a8 [pagezero] 18 0 0 0 DL psleep 0xc1a3f124 [vmdaemon] 17 0 0 0 DL (threaded) [pagedaemon] 100081 D psleep 0xc1a9c104 [pagedaemon] 100088 D umarcl 0xc1a3ed20 [uma] 16 0 0 0 DL - 0xc1a3859c [soaiod4] 15 0 0 0 DL - 0xc1a3859c [soaiod3] 9 0 0 0 DL - 0xc1a3859c [soaiod2] 8 0 0 0 DL - 0xc1a3859c [soaiod1] 7 0 0 0 DL waiting_ 0xc1a97a54 [sctp_iterator] 6 0 0 0 DL - 0xc9055e3c [fdc0] 14 0 0 0 DL (threaded) [usb] 100035 D - 0xc904aacc [usbus0] 100036 D - 0xc904aafc [usbus0] 100037 D - 0xc904ab2c [usbus0] 100038 D - 0xc904ab5c [usbus0] 100039 D - 0xc904ab8c [usbus0] 100041 D - 0xc909facc [usbus1] 100042 D - 0xc909fafc [usbus1] 100043 D - 0xc909fb2c [usbus1] 100044 D - 0xc909fb5c [usbus1] 100045 D - 0xc909fb8c [usbus1] 100047 D - 0xc90a8acc [usbus2] 100048 D - 0xc90a8afc [usbus2] 100049 D - 0xc90a8b2c [usbus2] 100050 D - 0xc90a8b5c [usbus2] 100051 D - 0xc90a8b8c [usbus2] 100052 D - 0xc90aeacc [usbus3] 100053 D - 0xc90aeafc [usbus3] 100054 D - 0xc90aeb2c [usbus3] 100055 D - 0xc90aeb5c [usbus3] 100056 D - 0xc90aeb8c [usbus3] 100057 D - 0xc90b8acc [usbus4] 100058 D - 0xc90b8afc [usbus4] 100059 D - 0xc90b8b2c [usbus4] 100060 D - 0xc90b8b5c [usbus4] 100061 D - 0xc90b8b8c [usbus4] 100063 D - 0xc904fca4 [usbus5] 100064 D - 0xc904fcd4 [usbus5] 100065 D - 0xc904fd04 [usbus5] 100066 D - 0xc904fd34 [usbus5] 100067 D - 0xc904fd64 [usbus5] 5 0 0 0 DL (threaded) [cam] 100021 D - 0xc1861f00 [doneq0] 100076 D - 0xc1861dac [scanner] 4 0 0 0 DL crypto_r 0xc1a3e078 [crypto returns] 3 0 0 0 DL crypto_w 0xc1a3dfc0 [crypto] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc1a92bec [g_event] 100016 D - 0xc1a92bf0 [g_up] 100017 D - 0xc1a92bf4 [g_down] 12 0 0 0 WL (threaded) [intr] 100008 I [swi4: clock (0)] 100009 I [swi4: clock (1)] 100010 I [swi4: clock (2)] 100011 I [swi4: clock (3)] 100012 I [swi3: vm] 100013 I [swi1: netisr 0] 100026 I [swi6: task queue] 100028 I [swi6: Giant taskq] 100030 I [swi5: fast taskq] 100031 I [irq256: hdac0] 100032 I [irq257: re0] 100033 I [irq22: ahci0] 100034 I [irq16: hdac1 ohci0] 100040 I [irq17: ohci1 ohci3] 100046 I [irq18: ohci2 ohci4] 100062 I [irq19: ehci0] 100068 I [irq14: ata0] 100069 I [swi0: uart] 100071 I [irq1: atkbd0] 100072 I [irq12: psm0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100006 CanRun [idle: cpu2] 100007 CanRun [idle: cpu3] 2 0 0 0 RL [rand_harvestq] 1 0 1 0 SLs wait 0xc8e44a08 [init] 10 0 0 0 DL audit_wo 0xc1a9a94c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xc1a92c14 [swapper] 100014 D - 0xc8f0f000 [firmware taskq] 100020 D - 0xc8f0ef00 [aiod_kick taskq] 100022 D - 0xc8f0ed80 [kqueue taskq] 100023 D - 0xc8f0ed00 [acpi_task_0] 100024 D - 0xc8f0ed00 [acpi_task_1] 100025 D - 0xc8f0ed00 [acpi_task_2] 100027 D - 0xc8f0ec00 [ffs_trim taskq] 100029 D - 0xc8f0eb00 [thread taskq] 100073 D - 0xc8f0e600 [mca taskq] 100075 D - 0xc8f0ee80 [CAM taskq] 2020 1065 960 0 Z+ mount db> allt Tracing command date pid 2039 tid 100257 td 0xcbd6b340 sched_switch(cbd6b340,0,103,1b0,c19dd558,...) at sched_switch+0x4da/frame 0xf3aa66b8 mi_switch(103,0,c15847f0,2e5,c8cdef00,...) at mi_switch+0x1b4/frame 0xf3aa66f0 turnstile_wait(c8cdef00,0,0,c89a8fa8,c1a0999c,...) at turnstile_wait+0x92b/frame 0xf3aa67b0 __rw_wlock_hard(c96ca6f0,cbd6b340,c15c79dc,79,0,...) at __rw_wlock_hard+0x370/frame 0xf3aa6824 _rw_wlock_cookie(c96ca6f0,c15c79dc,79,f3aa6898,c18295d8,...) at _rw_wlock_cookie+0xdd/frame 0xf3aa6858 vnode_create_vobject(c9a0fca8,1e,0,cbd6b340,c9a0fca8,...) at vnode_create_vobject+0x120/frame 0xf3aa6914 ufs_open(f3aa69d8,c17706ce,f3aa6950,c0f1e48f,cbde3400,...) at ufs_open+0x70/frame 0xf3aa6934 VOP_OPEN_APV(c1849494,f3aa69d8,100,f3aa6b98,cbd6b340,...) at VOP_OPEN_APV+0x21e/frame 0xf3aa6990 vn_open_vnode(c9a0fca8,100001,c95f3c00,cbd6b340,c9758540,...) at vn_open_vnode+0x1f9/frame 0xf3aa6a08 vn_open_cred(f3aa6b48,f3aa6bd4,0,0,c95f3c00,c9758540) at vn_open_cred+0x33f/frame 0xf3aa6ad8 vn_open(f3aa6b48,f3aa6bd4,0,c9758540,28073000,...) at vn_open+0x3d/frame 0xf3aa6b00 kern_openat(cbd6b340,ffffff9c,28073000,0,100000,0) at kern_openat+0x361/frame 0xf3aa6bf8 sys_openat(cbd6b340,f3aa6ca8,c1574eec,4,0,...) at sys_openat+0x3a/frame 0xf3aa6c20 syscall(f3aa6ce8) at syscall+0x388/frame 0xf3aa6cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3aa6cdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2805c55b, esp = 0xbfbfd5dc, ebp = 0xbfbfd600 --- Tracing command date pid 2038 tid 100197 td 0xca18b680 cpustop_handler(2,13,f39b293c,c1196a27,c8ffc920,...) at cpustop_handler+0x25/frame 0xf39b27e8 ipi_nmi_handler(c8ffc920,c181dc00,0,0,0,...) at ipi_nmi_handler+0x36/frame 0xf39b27f8 trap(f39b2948) at trap+0x37/frame 0xf39b293c calltrap() at calltrap+0x6/frame 0xf39b293c --- trap 0x13, eip = 0xc0c2c5cd, esp = 0xf39b2988, ebp = 0xf39b29f4 --- __rw_wlock_hard(c1a51f10,ca18b680,c1763cf5,d89,0,...) at __rw_wlock_hard+0xfd/frame 0xf39b29f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,ca18b680,...) at _rw_wlock_cookie+0xdd/frame 0xf39b2a28 pmap_enter(ca0fe384,28067000,c5c763fc,3,2,...) at pmap_enter+0x7c/frame 0xf39b2a78 vm_fault_hold(ca0fe2d0,28067000,2,0,0,...) at vm_fault_hold+0x17da/frame 0xf39b2b28 vm_fault(ca0fe2d0,28067000,2,0,3,...) at vm_fault+0x90/frame 0xf39b2b50 trap_pfault(28067ffc,f39b2c04,8000,3,a233ffc0,...) at trap_pfault+0x144/frame 0xf39b2b98 trap(f39b2ce8) at trap+0x591/frame 0xf39b2cdc calltrap() at calltrap+0x6/frame 0xf39b2cdc --- trap 0xc, eip = 0x280590a4, esp = 0xbfbfe364, ebp = 0xbfbfe380 --- Tracing command ls pid 2037 tid 100238 td 0xcbd6c000 sched_switch(cbd6c000,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3a62888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3a628c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a09a4c,...) at turnstile_wait+0x92b/frame 0xf3a62980 __rw_wlock_hard(c1a51f10,cbd6c000,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xf3a629f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbd6c000,...) at _rw_wlock_cookie+0xdd/frame 0xf3a62a28 pmap_enter(ca0fd474,2806f000,c5c780f0,3,2,...) at pmap_enter+0x7c/frame 0xf3a62a78 vm_fault_hold(ca0fd3c0,2806f000,2,0,0,...) at vm_fault_hold+0x17da/frame 0xf3a62b28 vm_fault(ca0fd3c0,2806f000,2,0,0,...) at vm_fault+0x90/frame 0xf3a62b50 trap_pfault(2806f03c,cbe51417,0,0,a233ffc0,...) at trap_pfault+0x144/frame 0xf3a62b98 trap(f3a62ce8) at trap+0x591/frame 0xf3a62cdc calltrap() at calltrap+0x6/frame 0xf3a62cdc --- trap 0xc, eip = 0x2805b0a4, esp = 0xbfbfd3b4, ebp = 0xbfbfd3d0 --- Tracing command umount pid 2036 tid 100203 td 0xcaecf680 sched_switch(caecf680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf39c47d0 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf39c4808 turnstile_wait(c9739a80,cbe3b340,0,c89a02b0,c1a0a81c,...) at turnstile_wait+0x92b/frame 0xf39c48c8 __rw_wlock_hard(c1a51f10,caecf680,c1763cf5,e91,0,...) at __rw_wlock_hard+0x370/frame 0xf39c493c _rw_wlock_cookie(c1a51f10,c1763cf5,e91,e8c,0,...) at _rw_wlock_cookie+0xdd/frame 0xf39c4970 pmap_enter_object(c96d9924,2806e000,2806f000,c2a92804,1,...) at pmap_enter_object+0x51/frame 0xf39c49b0 vm_map_pmap_enter(1,c96b01e0,0,0,1000,...) at vm_map_pmap_enter+0x372/frame 0xf39c4a0c vm_map_insert(c96d9870,c96b01e0,0,0,2806e000,...) at vm_map_insert+0x595/frame 0xf39c4a64 vm_map_find(c96d9870,c96b01e0,0,0,f39c4c04,...) at vm_map_find+0x220/frame 0xf39c4ac8 vm_mmap_object(c96d9870,f39c4c04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf39c4b20 vn_mmap(c9701000,c96d9870,f39c4c04,1000,1,...) at vn_mmap+0x158/frame 0xf39c4b98 sys_mmap(caecf680,f39c4ca8,c1574eec,4,f39c4c54,...) at sys_mmap+0x438/frame 0xf39c4c20 syscall(f39c4ce8) at syscall+0x388/frame 0xf39c4cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39c4cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2805c4eb, esp = 0xbfbfdc4c, ebp = 0xbfbfdcf0 --- Tracing command ls pid 2035 tid 100251 td 0xcbdaf000 sched_switch(cbdaf000,0,103,1b0,c19dd558,...) at sched_switch+0x4da/frame 0xf3a936b8 mi_switch(103,0,c15847f0,2e5,c8cdef00,...) at mi_switch+0x1b4/frame 0xf3a936f0 turnstile_wait(c8cdef00,0,0,c89a8fa8,c1a0a96c,...) at turnstile_wait+0x92b/frame 0xf3a937b0 __rw_wlock_hard(c96ca6f0,cbdaf000,c15c79dc,79,0,...) at __rw_wlock_hard+0x370/frame 0xf3a93824 _rw_wlock_cookie(c96ca6f0,c15c79dc,79,f3a93898,c18295d8,...) at _rw_wlock_cookie+0xdd/frame 0xf3a93858 vnode_create_vobject(c9a0fca8,1e,0,cbdaf000,c9a0fca8,...) at vnode_create_vobject+0x120/frame 0xf3a93914 ufs_open(f3a939d8,c17706ce,f3a93950,c0f1e48f,c967ac00,...) at ufs_open+0x70/frame 0xf3a93934 VOP_OPEN_APV(c1849494,f3a939d8,100,f3a93b98,cbdaf000,...) at VOP_OPEN_APV+0x21e/frame 0xf3a93990 vn_open_vnode(c9a0fca8,100001,c95f3c00,cbdaf000,c9780428,...) at vn_open_vnode+0x1f9/frame 0xf3a93a08 vn_open_cred(f3a93b48,f3a93bd4,0,0,c95f3c00,c9780428) at vn_open_cred+0x33f/frame 0xf3a93ad8 vn_open(f3a93b48,f3a93bd4,0,c9780428,28075000,...) at vn_open+0x3d/frame 0xf3a93b00 kern_openat(cbdaf000,ffffff9c,28075000,0,100000,0) at kern_openat+0x361/frame 0xf3a93bf8 sys_openat(cbdaf000,f3a93ca8,c1574eec,4,c119f5d9,...) at sys_openat+0x3a/frame 0xf3a93c20 syscall(f3a93ce8) at syscall+0x388/frame 0xf3a93cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a93cdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2805e55b, esp = 0xbfbfd53c, ebp = 0xbfbfd560 --- Tracing command grep pid 2034 tid 100250 td 0xcbdaf340 sched_switch(cbdaf340,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3a8f7d0 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3a8f808 turnstile_wait(c9739a80,cbe3b340,0,c89a02b0,c1a0a3fc,...) at turnstile_wait+0x92b/frame 0xf3a8f8c8 __rw_wlock_hard(c1a51f10,cbdaf340,c1763cf5,e91,0,...) at __rw_wlock_hard+0x370/frame 0xf3a8f93c _rw_wlock_cookie(c1a51f10,c1763cf5,e91,e8c,0,...) at _rw_wlock_cookie+0xdd/frame 0xf3a8f970 pmap_enter_object(c97bc474,2807f000,28080000,c2b7de50,1,...) at pmap_enter_object+0x51/frame 0xf3a8f9b0 vm_map_pmap_enter(1,c96ca6e0,0,0,1000,...) at vm_map_pmap_enter+0x372/frame 0xf3a8fa0c vm_map_insert(c97bc3c0,c96ca6e0,0,0,2807f000,...) at vm_map_insert+0x595/frame 0xf3a8fa64 vm_map_find(c97bc3c0,c96ca6e0,0,0,f3a8fc04,...) at vm_map_find+0x220/frame 0xf3a8fac8 vm_mmap_object(c97bc3c0,f3a8fc04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf3a8fb20 vn_mmap(c97593f0,c97bc3c0,f3a8fc04,1000,1,...) at vn_mmap+0x158/frame 0xf3a8fb98 sys_mmap(cbdaf340,f3a8fca8,c1574eec,4,f3a8fc54,...) at sys_mmap+0x438/frame 0xf3a8fc20 syscall(f3a8fce8) at syscall+0x388/frame 0xf3a8fcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a8fcdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2806d4eb, esp = 0xbfbfd5f4, ebp = 0xbfbfd690 --- Tracing command grep pid 2033 tid 100178 td 0xca3d69c0 sched_switch(ca3d69c0,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3972840 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3972878 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a09d0c,...) at turnstile_wait+0x92b/frame 0xf3972938 __rw_wlock_hard(c1a51f10,ca3d69c0,c1763cf5,b75,0,...) at __rw_wlock_hard+0x370/frame 0xf39729ac _rw_wlock_cookie(c1a51f10,c1763cf5,b75,2809e000,cbde5000,...) at _rw_wlock_cookie+0xdd/frame 0xf39729e0 pmap_remove(c97bcce4,2809e000,2809f000,ba7,0,...) at pmap_remove+0x40/frame 0xf3972a24 vm_map_delete(c97bcc30,2809e000,2809f000,5b2,c9a96ac0,...) at vm_map_delete+0x148/frame 0xf3972a88 vm_map_fixed(c97bcc30,c95c4c80,12000,0,2809e000,...) at vm_map_fixed+0xb8/frame 0xf3972ac8 vm_mmap_object(c97bcc30,f3972c04,1000,3,7,...) at vm_mmap_object+0x342/frame 0xf3972b20 vn_mmap(c97802d8,c97bcc30,f3972c04,1000,3,...) at vn_mmap+0x158/frame 0xf3972b98 sys_mmap(ca3d69c0,f3972ca8,c1574eec,4,c181dbf4,...) at sys_mmap+0x438/frame 0xf3972c20 syscall(f3972ce8) at syscall+0x388/frame 0xf3972cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3972cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2806d4eb, esp = 0xbfbfdc3c, ebp = 0xbfbfdcf0 --- Tracing command sh pid 2032 tid 100214 td 0xcbd8a000 sched_switch(cbd8a000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a089e4 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3a08a1c sleepq_switch(cbd8a000,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a08a50 sleepq_catch_signals(c1583a69,16f,14,1e0,cbd8a000,...) at sleepq_catch_signals+0x3b7/frame 0xf3a08a8c sleepq_wait_sig(cbdda000,5c,c1587368,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a08ab0 _sleep(cbdda000,cbdda180,15c,c1587368,0,...) at _sleep+0x37e/frame 0xf3a08b10 pipe_read(c9758310,f3a08bf0,c95f3c00,0,cbd8a000,...) at pipe_read+0x2ca/frame 0xf3a08b54 dofileread(c9758310,f3a08bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3a08b88 kern_readv(cbd8a000,3,f3a08bf0,c181dbf4,bfbfdef8,...) at kern_readv+0x93/frame 0xf3a08bd0 sys_read(cbd8a000,f3a08ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xf3a08c20 syscall(f3a08ce8) at syscall+0x388/frame 0xf3a08cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a08cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28204553, esp = 0xbfbfde5c, ebp = 0xbfbfde70 --- Tracing command mount pid 2031 tid 100275 td 0xcbe3c340 sched_switch(cbe3c340,0,103,1b0,c19dd2b8,...) at sched_switch+0x4da/frame 0xf3ae16b8 mi_switch(103,0,c15847f0,2e5,c8cde400,...) at mi_switch+0x1b4/frame 0xf3ae16f0 turnstile_wait(c8cde400,0,0,c89a8fa8,c1a09f74,...) at turnstile_wait+0x92b/frame 0xf3ae17b0 __rw_wlock_hard(c96b0a10,cbe3c340,c15c79dc,79,0,...) at __rw_wlock_hard+0x370/frame 0xf3ae1824 _rw_wlock_cookie(c96b0a10,c15c79dc,79,f3ae1898,c18295d8,...) at _rw_wlock_cookie+0xdd/frame 0xf3ae1858 vnode_create_vobject(c9685e10,66,0,cbe3c340,c9685e10,...) at vnode_create_vobject+0x120/frame 0xf3ae1914 ufs_open(f3ae19d8,c17706ce,f3ae1950,c0f1e48f,cbde3800,...) at ufs_open+0x70/frame 0xf3ae1934 VOP_OPEN_APV(c1849494,f3ae19d8,100,f3ae1b98,cbe3c340,...) at VOP_OPEN_APV+0x21e/frame 0xf3ae1990 vn_open_vnode(c9685e10,100001,c95f3c00,cbe3c340,c9780888,...) at vn_open_vnode+0x1f9/frame 0xf3ae1a08 vn_open_cred(f3ae1b48,f3ae1bd4,0,0,c95f3c00,c9780888) at vn_open_cred+0x33f/frame 0xf3ae1ad8 vn_open(f3ae1b48,f3ae1bd4,0,c9780888,2806d000,...) at vn_open+0x3d/frame 0xf3ae1b00 kern_openat(cbe3c340,ffffff9c,2806d000,0,100000,0) at kern_openat+0x361/frame 0xf3ae1bf8 sys_openat(cbe3c340,f3ae1ca8,c1574eec,4,0,...) at sys_openat+0x3a/frame 0xf3ae1c20 syscall(f3ae1ce8) at syscall+0x388/frame 0xf3ae1cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ae1cdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2805d55b, esp = 0xbfbfe2dc, ebp = 0xbfbfe300 --- Tracing command grep pid 2030 tid 100278 td 0xcbe9b000 sched_switch(cbe9b000,0,103,1b0,c19dd558,...) at sched_switch+0x4da/frame 0xf3aea868 mi_switch(103,0,c15847f0,2e5,c8cdef00,...) at mi_switch+0x1b4/frame 0xf3aea8a0 turnstile_wait(c8cdef00,0,0,c89a0248,c89a0248,...) at turnstile_wait+0x92b/frame 0xf3aea960 __rw_wlock_hard(c96ca6f0,cbe9b000,c15c35bb,4ee,0,...) at __rw_wlock_hard+0x370/frame 0xf3aea9d8 _rw_wlock_cookie(c96ca6f0,c15c35bb,4ee,4a3,8,...) at _rw_wlock_cookie+0xdd/frame 0xf3aeaa0c vm_map_insert(ca0fe780,c96ca6e0,0,0,2807f000,...) at vm_map_insert+0x26d/frame 0xf3aeaa64 vm_map_find(ca0fe780,c96ca6e0,0,0,f3aeac04,...) at vm_map_find+0x220/frame 0xf3aeaac8 vm_mmap_object(ca0fe780,f3aeac04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf3aeab20 vn_mmap(c97592d8,ca0fe780,f3aeac04,1000,1,...) at vn_mmap+0x158/frame 0xf3aeab98 sys_mmap(cbe9b000,f3aeaca8,c1574eec,4,c181dbf4,...) at sys_mmap+0x438/frame 0xf3aeac20 syscall(f3aeace8) at syscall+0x388/frame 0xf3aeacdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3aeacdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2806d4eb, esp = 0xbfbfd5f4, ebp = 0xbfbfd690 --- Tracing command umount pid 2029 tid 100265 td 0xcbe71680 sched_switch(cbe71680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3ac37d0 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3ac3808 turnstile_wait(c9739a80,cbe3b340,0,c89a02b0,c1a09dcc,...) at turnstile_wait+0x92b/frame 0xf3ac38c8 __rw_wlock_hard(c1a51f10,cbe71680,c1763cf5,e91,0,...) at __rw_wlock_hard+0x370/frame 0xf3ac393c _rw_wlock_cookie(c1a51f10,c1763cf5,e91,e8c,0,...) at _rw_wlock_cookie+0xdd/frame 0xf3ac3970 pmap_enter_object(c8e51ce4,2806f000,28070000,c2a91f40,1,...) at pmap_enter_object+0x51/frame 0xf3ac39b0 vm_map_pmap_enter(1,c96b0a00,0,0,1000,...) at vm_map_pmap_enter+0x372/frame 0xf3ac3a0c vm_map_insert(c8e51c30,c96b0a00,0,0,2806f000,...) at vm_map_insert+0x595/frame 0xf3ac3a64 vm_map_find(c8e51c30,c96b0a00,0,0,f3ac3c04,...) at vm_map_find+0x220/frame 0xf3ac3ac8 vm_mmap_object(c8e51c30,f3ac3c04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf3ac3b20 vn_mmap(c97589d8,c8e51c30,f3ac3c04,1000,1,...) at vn_mmap+0x158/frame 0xf3ac3b98 sys_mmap(cbe71680,f3ac3ca8,c1574eec,4,f3ac3c54,...) at sys_mmap+0x438/frame 0xf3ac3c20 syscall(f3ac3ce8) at syscall+0x388/frame 0xf3ac3cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ac3cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2805c4eb, esp = 0xbfbfe2e4, ebp = 0xbfbfe37c --- Tracing command sh pid 2028 tid 100253 td 0xcbe3a9c0 sched_switch(cbe3a9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a9a9e4 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3a9aa1c sleepq_switch(cbe3a9c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf3a9aa50 sleepq_catch_signals(c1583a69,16f,14,150,cbe3a9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf3a9aa8c sleepq_wait_sig(c97b7000,5c,c1587368,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a9aab0 _sleep(c97b7000,c97b7180,15c,c1587368,0,...) at _sleep+0x37e/frame 0xf3a9ab10 pipe_read(c9746e38,f3a9abf0,c95f3c00,0,cbe3a9c0,...) at pipe_read+0x2ca/frame 0xf3a9ab54 dofileread(c9746e38,f3a9abf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3a9ab88 kern_readv(cbe3a9c0,3,f3a9abf0,cbe366b0,bfbfdef8,...) at kern_readv+0x93/frame 0xf3a9abd0 sys_read(cbe3a9c0,f3a9aca8,c1574eec,4,0,...) at sys_read+0x59/frame 0xf3a9ac20 syscall(f3a9ace8) at syscall+0x388/frame 0xf3a9acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a9acdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28204553, esp = 0xbfbfde5c, ebp = 0xbfbfde70 --- Tracing command mount pid 2027 tid 100277 td 0xcbe9b340 sched_switch(cbe9b340,0,103,1b0,c19dd558,...) at sched_switch+0x4da/frame 0xf3ae76b8 mi_switch(103,0,c15847f0,2e5,c8cdef00,...) at mi_switch+0x1b4/frame 0xf3ae76f0 turnstile_wait(c8cdef00,0,0,c89a8fa8,c1a0a0d4,...) at turnstile_wait+0x92b/frame 0xf3ae77b0 __rw_wlock_hard(c96ca6f0,cbe9b340,c15c79dc,79,0,...) at __rw_wlock_hard+0x370/frame 0xf3ae7824 _rw_wlock_cookie(c96ca6f0,c15c79dc,79,f3ae7898,c18295d8,...) at _rw_wlock_cookie+0xdd/frame 0xf3ae7858 vnode_create_vobject(c9a0fca8,1e,0,cbe9b340,c9a0fca8,...) at vnode_create_vobject+0x120/frame 0xf3ae7914 ufs_open(f3ae79d8,c17706ce,f3ae7950,c0f1e48f,c9754c00,...) at ufs_open+0x70/frame 0xf3ae7934 VOP_OPEN_APV(c1849494,f3ae79d8,100,f3ae7b98,cbe9b340,...) at VOP_OPEN_APV+0x21e/frame 0xf3ae7990 vn_open_vnode(c9a0fca8,100001,c95f3c00,cbe9b340,c9780460,...) at vn_open_vnode+0x1f9/frame 0xf3ae7a08 vn_open_cred(f3ae7b48,f3ae7bd4,0,0,c95f3c00,c9780460) at vn_open_cred+0x33f/frame 0xf3ae7ad8 vn_open(f3ae7b48,f3ae7bd4,0,c9780460,28074000,...) at vn_open+0x3d/frame 0xf3ae7b00 kern_openat(cbe9b340,ffffff9c,28074000,0,100000,0) at kern_openat+0x361/frame 0xf3ae7bf8 sys_openat(cbe9b340,f3ae7ca8,c1574eec,4,f3ae7c54,...) at sys_openat+0x3a/frame 0xf3ae7c20 syscall(f3ae7ce8) at syscall+0x388/frame 0xf3ae7cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ae7cdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2805d55b, esp = 0xbfbfd5ec, ebp = 0xbfbfd610 --- Tracing command ls pid 2026 tid 100194 td 0xca18b9c0 sched_switch(ca18b9c0,0,103,1b0,c19dd558,...) at sched_switch+0x4da/frame 0xf39ac9d8 mi_switch(103,0,c15847f0,2e5,c8cdef00,...) at mi_switch+0x1b4/frame 0xf39aca10 turnstile_wait(c8cdef00,0,0,c89a0248,c89a0248,...) at turnstile_wait+0x92b/frame 0xf39acad0 __rw_wlock_hard(c96ca6f0,ca18b9c0,c15c35bb,b73,0,...) at __rw_wlock_hard+0x370/frame 0xf39acb48 _rw_wlock_cookie(c96ca6f0,c15c35bb,b73,ba7,c0c39d6d,...) at _rw_wlock_cookie+0xdd/frame 0xf39acb7c vm_map_delete(c973d960,28070000,28071000,4,1000,...) at vm_map_delete+0x206/frame 0xf39acbe0 sys_munmap(ca18b9c0,f39acca8,c1574eec,4,c181dbf4,...) at sys_munmap+0x11a/frame 0xf39acc20 syscall(f39acce8) at syscall+0x388/frame 0xf39accdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39accdc --- syscall (73, FreeBSD ELF32, sys_munmap), eip = 0x2805e42b, esp = 0xbfbfd564, ebp = 0xbfbfd600 --- Tracing command mount pid 2025 tid 100248 td 0xcbe3b340 cpustop_handler(1,13,f3a875c4,c1196a27,f3a874e0,...) at cpustop_handler+0x25/frame 0xf3a87470 ipi_nmi_handler(f3a874e0,f3a874e4,c0bee139,c89a02b0,f3a874f0,...) at ipi_nmi_handler+0x36/frame 0xf3a87480 trap(f3a875d0) at trap+0x37/frame 0xf3a875c4 calltrap() at calltrap+0x6/frame 0xf3a875c4 --- trap 0x13, eip = 0xc11c19e2, esp = 0xf3a87610, ebp = 0xf3a87648 --- DELAY(1,4,cbe3b340,38e,0,...) at DELAY+0x152/frame 0xf3a87648 thread_lock_flags_(caece340,0,c1583a69,38e,f3a87800,...) at thread_lock_flags_+0x54c/frame 0xf3a8775c sleepq_timeout(caece340,0,c157ed51,2ab,1bb,...) at sleepq_timeout+0x2c/frame 0xf3a87780 softclock_call_cc(1,2,c157ed51,1bb,c19cd380,...) at softclock_call_cc+0x1d9/frame 0xf3a87820 callout_process(eba8f2f9,140,c17667b7,df,f3a87948,...) at callout_process+0x236/frame 0xf3a87894 handleevents(0,f3a87940,c0c10de1,c181dacc,f3a87918,...) at handleevents+0x25c/frame 0xf3a878d8 timercb(c1a88604,0,1,1,0,...) at timercb+0x27f/frame 0xf3a8791c lapic_handle_timer(f3a87948) at lapic_handle_timer+0x7c/frame 0xf3a8793c Xtimerint() at Xtimerint+0x2e/frame 0xf3a8793c --- interrupt, eip = 0xc0c2c5d7, esp = 0xf3a87988, ebp = 0xf3a879f4 --- __rw_wlock_hard(c1a51f10,cbe3b340,c1763cf5,d89,0,...) at __rw_wlock_hard+0x107/frame 0xf3a879f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbe3b340,...) at _rw_wlock_cookie+0xdd/frame 0xf3a87a28 pmap_enter(c97bcbf4,28401000,c32f5000,3,2,...) at pmap_enter+0x7c/frame 0xf3a87a78 vm_fault_hold(c97bcb40,28401000,2,0,0,...) at vm_fault_hold+0x17da/frame 0xf3a87b28 vm_fault(c97bcb40,28401000,2,0,f3a87bc0,...) at vm_fault+0x90/frame 0xf3a87b50 trap_pfault(28401000,c181dc00,0,0,0,...) at trap_pfault+0x144/frame 0xf3a87b98 trap(f3a87ce8) at trap+0x591/frame 0xf3a87cdc calltrap() at calltrap+0x6/frame 0xf3a87cdc --- trap 0xc, eip = 0x28132b7e, esp = 0xbfbfda90, ebp = 0xbfbfdab0 --- Tracing command umount pid 2024 tid 100090 td 0xc973c340 sched_switch(c973c340,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xedf26888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xedf268c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a097e4,...) at turnstile_wait+0x92b/frame 0xedf26980 __rw_wlock_hard(c1a51f10,c973c340,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xedf269f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,c973c340,...) at _rw_wlock_cookie+0xdd/frame 0xedf26a28 pmap_enter(c97bace4,2806d000,c5c78288,3,2,...) at pmap_enter+0x7c/frame 0xedf26a78 vm_fault_hold(c97bac30,2806d000,2,0,0,...) at vm_fault_hold+0x17da/frame 0xedf26b28 vm_fault(c97bac30,2806d000,2,0,0,...) at vm_fault+0x90/frame 0xedf26b50 trap_pfault(2806d03c,c9755017,0,0,a233ffc0,...) at trap_pfault+0x144/frame 0xedf26b98 trap(edf26ce8) at trap+0x591/frame 0xedf26cdc calltrap() at calltrap+0x6/frame 0xedf26cdc --- trap 0xc, eip = 0x280590a4, esp = 0xbfbfd444, ebp = 0xbfbfd460 --- Tracing command umount pid 2023 tid 100237 td 0xcbd6c340 sched_switch(cbd6c340,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3a5f888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3a5f8c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a09bac,...) at turnstile_wait+0x92b/frame 0xf3a5f980 __rw_wlock_hard(c1a51f10,cbd6c340,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xf3a5f9f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbd6c340,...) at _rw_wlock_cookie+0xdd/frame 0xf3a5fa28 pmap_enter(ca0fdbf4,2841f000,c55067f8,3,1,...) at pmap_enter+0x7c/frame 0xf3a5fa78 vm_fault_hold(ca0fdb40,2841f000,1,0,0,...) at vm_fault_hold+0x17da/frame 0xf3a5fb28 vm_fault(ca0fdb40,2841f000,1,0,c19ccc90,...) at vm_fault+0x90/frame 0xf3a5fb50 trap_pfault(2841f000,4,c19ccc80,c19ccc90,c19ccc80,...) at trap_pfault+0x144/frame 0xf3a5fb98 trap(f3a5fce8) at trap+0x591/frame 0xf3a5fcdc calltrap() at calltrap+0x6/frame 0xf3a5fcdc --- trap 0xc, eip = 0x28122250, esp = 0xbfbfe00c, ebp = 0xbfbfe044 --- Tracing command grep pid 2022 tid 100232 td 0xcbd6c680 sched_switch(cbd6c680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3a4a888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3a4a8c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a0983c,...) at turnstile_wait+0x92b/frame 0xf3a4a980 __rw_wlock_hard(c1a51f10,cbd6c680,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xf3a4a9f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbd6c680,...) at _rw_wlock_cookie+0xdd/frame 0xf3a4aa28 pmap_enter(ca0fe474,2862a000,c3daf2e4,3,1,...) at pmap_enter+0x7c/frame 0xf3a4aa78 vm_fault_hold(ca0fe3c0,2862a000,1,0,0,...) at vm_fault_hold+0x17da/frame 0xf3a4ab28 vm_fault(ca0fe3c0,2862a000,1,0,2,...) at vm_fault+0x90/frame 0xf3a4ab50 trap_pfault(2862a000,f3a4ace8,f3a4abb8,bfbfe2e0,a233ffc0,...) at trap_pfault+0x144/frame 0xf3a4ab98 trap(f3a4ace8) at trap+0x591/frame 0xf3a4acdc calltrap() at calltrap+0x6/frame 0xf3a4acdc --- trap 0xc, eip = 0x2816b250, esp = 0xbfbfe0fc, ebp = 0xbfbfe134 --- Tracing command grep pid 2021 tid 100273 td 0xcbe3c9c0 sched_switch(cbe3c9c0,0,103,1b0,c19dd8e8,...) at sched_switch+0x4da/frame 0xf3adb6b8 mi_switch(103,0,c15847f0,2e5,c952f680,...) at mi_switch+0x1b4/frame 0xf3adb6f0 turnstile_wait(c952f680,0,0,c89a8fa8,c1a09aa4,...) at turnstile_wait+0x92b/frame 0xf3adb7b0 __rw_wlock_hard(c95c4c90,cbe3c9c0,c15c79dc,79,0,...) at __rw_wlock_hard+0x370/frame 0xf3adb824 _rw_wlock_cookie(c95c4c90,c15c79dc,79,f3adb898,c18295d8,...) at _rw_wlock_cookie+0xdd/frame 0xf3adb858 vnode_create_vobject(c9b819d8,13174,0,cbe3c9c0,c9b819d8,...) at vnode_create_vobject+0x120/frame 0xf3adb914 ufs_open(f3adb9d8,c17706ce,f3adb950,c0f1e48f,c967ac00,...) at ufs_open+0x70/frame 0xf3adb934 VOP_OPEN_APV(c1849494,f3adb9d8,8000100,f3adbb98,cbe3c9c0,...) at VOP_OPEN_APV+0x21e/frame 0xf3adb990 vn_open_vnode(c9b819d8,300001,c95f3c00,cbe3c9c0,c9780968,...) at vn_open_vnode+0x1f9/frame 0xf3adba08 vn_open_cred(f3adbb48,f3adbbd4,0,0,c95f3c00,c9780968) at vn_open_cred+0x33f/frame 0xf3adbad8 vn_open(f3adbb48,f3adbbd4,0,c9780968,280790a0,...) at vn_open+0x3d/frame 0xf3adbb00 kern_openat(cbe3c9c0,ffffff9c,280790a0,0,300000,0) at kern_openat+0x361/frame 0xf3adbbf8 sys_openat(cbe3c9c0,f3adbca8,c1574eec,4,f3adbc54,...) at sys_openat+0x3a/frame 0xf3adbc20 syscall(f3adbce8) at syscall+0x388/frame 0xf3adbcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3adbcdc --- syscall (499, FreeBSD ELF32, sys_openat), eip = 0x2806d55b, esp = 0xbfbfdccc, ebp = 0xbfbfdcf0 --- Tracing command mount pid 2019 tid 100276 td 0xcbe9b680 sched_switch(cbe9b680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3ae47d0 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3ae4808 turnstile_wait(c9739a80,cbe3b340,0,c89a02b0,c1a098fc,...) at turnstile_wait+0x92b/frame 0xf3ae48c8 __rw_wlock_hard(c1a51f10,cbe9b680,c1763cf5,e91,0,...) at __rw_wlock_hard+0x370/frame 0xf3ae493c _rw_wlock_cookie(c1a51f10,c1763cf5,e91,e8c,0,...) at _rw_wlock_cookie+0xdd/frame 0xf3ae4970 pmap_enter_object(c973f564,2806f000,28070000,c2b78994,1,...) at pmap_enter_object+0x51/frame 0xf3ae49b0 vm_map_pmap_enter(1,c95c5dc0,0,0,1000,...) at vm_map_pmap_enter+0x372/frame 0xf3ae4a0c vm_map_insert(c973f4b0,c95c5dc0,0,0,2806f000,...) at vm_map_insert+0x595/frame 0xf3ae4a64 vm_map_find(c973f4b0,c95c5dc0,0,0,f3ae4c04,...) at vm_map_find+0x220/frame 0xf3ae4ac8 vm_mmap_object(c973f4b0,f3ae4c04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf3ae4b20 vn_mmap(c9759188,c973f4b0,f3ae4c04,1000,1,...) at vn_mmap+0x158/frame 0xf3ae4b98 sys_mmap(cbe9b680,f3ae4ca8,c1574eec,4,c181dbf4,...) at sys_mmap+0x438/frame 0xf3ae4c20 syscall(f3ae4ce8) at syscall+0x388/frame 0xf3ae4cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ae4cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2805d4eb, esp = 0xbfbfdc6c, ebp = 0xbfbfdd10 --- Tracing command grep pid 2018 tid 100271 td 0xcbe70340 sched_switch(cbe70340,0,103,1b0,c19dd8e8,...) at sched_switch+0x4da/frame 0xf3ad5890 mi_switch(103,0,c15847f0,2e5,c952f680,...) at mi_switch+0x1b4/frame 0xf3ad58c8 turnstile_wait(c952f680,0,0,c89a0248,c1a094cc,...) at turnstile_wait+0x92b/frame 0xf3ad5988 __rw_wlock_hard(c95c4c90,cbe70340,c15c35bb,4ee,0,...) at __rw_wlock_hard+0x370/frame 0xf3ad59fc _rw_wlock_cookie(c95c4c90,c15c35bb,4ee,4a3,cbe70340,...) at _rw_wlock_cookie+0xdd/frame 0xf3ad5a30 vm_map_insert(ca0fd780,c95c4c80,0,0,2808b000,...) at vm_map_insert+0x26d/frame 0xf3ad5a88 vm_map_fixed(ca0fd780,c95c4c80,0,0,2808b000,...) at vm_map_fixed+0x130/frame 0xf3ad5ac8 vm_mmap_object(ca0fd780,f3ad5c04,13000,5,7,...) at vm_mmap_object+0x342/frame 0xf3ad5b20 vn_mmap(c97586c8,ca0fd780,f3ad5c04,13000,5,...) at vn_mmap+0x158/frame 0xf3ad5b98 sys_mmap(cbe70340,f3ad5ca8,c1574eec,4,c119f5d9,...) at sys_mmap+0x438/frame 0xf3ad5c20 syscall(f3ad5ce8) at syscall+0x388/frame 0xf3ad5cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ad5cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2806d4eb, esp = 0xbfbfdc3c, ebp = 0xbfbfdcf0 --- Tracing command umount pid 2016 tid 100241 td 0xcbd6b680 sched_switch(cbd6b680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3a6e888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3a6e8c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a09b54,...) at turnstile_wait+0x92b/frame 0xf3a6e980 __rw_wlock_hard(c1a51f10,cbd6b680,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xf3a6e9f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbd6b680,...) at _rw_wlock_cookie+0xdd/frame 0xf3a6ea28 pmap_enter(c973f474,28206000,c4670468,3,2,...) at pmap_enter+0x7c/frame 0xf3a6ea78 vm_fault_hold(c973f3c0,28206000,2,0,0,...) at vm_fault_hold+0x17da/frame 0xf3a6eb28 vm_fault(c973f3c0,28206000,2,0,2,...) at vm_fault+0x90/frame 0xf3a6eb50 trap_pfault(28206340,f3a6ece8,c19cd380,c19cd390,c19cd380,...) at trap_pfault+0x144/frame 0xf3a6eb98 trap(f3a6ece8) at trap+0x591/frame 0xf3a6ecdc calltrap() at calltrap+0x6/frame 0xf3a6ecdc --- trap 0xc, eip = 0x28128d89, esp = 0xbfbfdef4, ebp = 0xbfbfdef4 --- Tracing command grep pid 2014 tid 100193 td 0xcaed0000 sched_switch(caed0000,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf39a97d0 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf39a9808 turnstile_wait(c9739a80,cbe3b340,0,c89a02b0,c1a09d74,...) at turnstile_wait+0x92b/frame 0xf39a98c8 __rw_wlock_hard(c1a51f10,caed0000,c1763cf5,e91,0,...) at __rw_wlock_hard+0x370/frame 0xf39a993c _rw_wlock_cookie(c1a51f10,c1763cf5,e91,e8c,0,...) at _rw_wlock_cookie+0xdd/frame 0xf39a9970 pmap_enter_object(ca0fd294,2807f000,28080000,c2b818c0,1,...) at pmap_enter_object+0x51/frame 0xf39a99b0 vm_map_pmap_enter(1,c95c4c80,0,0,1000,...) at vm_map_pmap_enter+0x372/frame 0xf39a9a0c vm_map_insert(ca0fd1e0,c95c4c80,0,0,2807f000,...) at vm_map_insert+0x595/frame 0xf39a9a64 vm_map_find(ca0fd1e0,c95c4c80,0,0,f39a9c04,...) at vm_map_find+0x220/frame 0xf39a9ac8 vm_mmap_object(ca0fd1e0,f39a9c04,1000,1,7,...) at vm_mmap_object+0x3a7/frame 0xf39a9b20 vn_mmap(c9758428,ca0fd1e0,f39a9c04,1000,1,...) at vn_mmap+0x158/frame 0xf39a9b98 sys_mmap(caed0000,f39a9ca8,c1574eec,4,f39a9c54,...) at sys_mmap+0x438/frame 0xf39a9c20 syscall(f39a9ce8) at syscall+0x388/frame 0xf39a9cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39a9cdc --- syscall (477, FreeBSD ELF32, sys_mmap), eip = 0x2806d4eb, esp = 0xbfbfdc4c, ebp = 0xbfbfdcf0 --- Tracing command mount pid 2013 tid 100272 td 0xcbe70000 cpustop_handler(3,13,f3ad8a5c,c1196a27,c11b871d,...) at cpustop_handler+0x25/frame 0xf3ad8908 ipi_nmi_handler(c11b871d,c1a88604,0,3,3,...) at ipi_nmi_handler+0x36/frame 0xf3ad8918 trap(f3ad8a68) at trap+0x37/frame 0xf3ad8a5c calltrap() at calltrap+0x6/frame 0xf3ad8a5c --- trap 0x13, eip = 0xc11be4d2, esp = 0xf3ad8aa8, ebp = 0xf3ad8ae0 --- smp_targeted_tlb_shootdown(0,c6cda000,c6cdb000) at smp_targeted_tlb_shootdown+0x242/frame 0xf3ad8ae0 smp_masked_invlpg_range(f,c6cda000,c6cdb000,c467a054,c4671678,...) at smp_masked_invlpg_range+0x31/frame 0xf3ad8af4 pmap_qremove(c6cda000,1,c6cda000,c6cda000,f3ad8b98,...) at pmap_qremove+0xa2/frame 0xf3ad8b10 free_pv_chunk(be3ee000,bfca1000,c1763cf5,1182,891,...) at free_pv_chunk+0x94/frame 0xf3ad8b28 pmap_remove_pages(c97bb924,cbe8cab0,cbe8cac0,cbe8cab0,cbe8ca08,...) at pmap_remove_pages+0x602/frame 0xf3ad8b98 vmspace_exit(cbe70000,0,c1574f40,161,cbe8ca08,...) at vmspace_exit+0x9b/frame 0xf3ad8bc4 exit1(cbe70000,0,0,f3ad8cdc,c1197f68,...) at exit1+0x67c/frame 0xf3ad8c0c sys_sys_exit(cbe70000,f3ad8ca8,c1574eec,4,f3ad8c54,...) at sys_sys_exit+0x22/frame 0xf3ad8c20 syscall(f3ad8ce8) at syscall+0x388/frame 0xf3ad8cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ad8cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x281611e3, esp = 0xbfbfe854, ebp = 0xbfbfe864 --- Tracing command ls pid 2001 tid 100274 td 0xcbe3c680 sched_switch(cbe3c680,0,103,1b0,c19dd4b0,...) at sched_switch+0x4da/frame 0xf3ade888 mi_switch(103,0,c15847f0,2e5,c9739a80,...) at mi_switch+0x1b4/frame 0xf3ade8c0 turnstile_wait(c9739a80,cbe3b340,0,c89a0248,c1a095d4,...) at turnstile_wait+0x92b/frame 0xf3ade980 __rw_wlock_hard(c1a51f10,cbe3c680,c1763cf5,d89,0,...) at __rw_wlock_hard+0x370/frame 0xf3ade9f4 _rw_wlock_cookie(c1a51f10,c1763cf5,d89,c181f99c,cbe3c680,...) at _rw_wlock_cookie+0xdd/frame 0xf3adea28 pmap_enter(c8e50564,2866a000,c36953e4,3,1,...) at pmap_enter+0x7c/frame 0xf3adea78 vm_fault_hold(c8e504b0,2866a000,1,0,0,...) at vm_fault_hold+0x17da/frame 0xf3adeb28 vm_fault(c8e504b0,2866a000,1,0,0,...) at vm_fault+0x90/frame 0xf3adeb50 trap_pfault(2866a000,c181dc00,200ff55,61a8e,a233ffc0,...) at trap_pfault+0x144/frame 0xf3adeb98 trap(f3adece8) at trap+0x591/frame 0xf3adecdc calltrap() at calltrap+0x6/frame 0xf3adecdc --- trap 0xc, eip = 0x28199250, esp = 0xbfbfdf5c, ebp = 0xbfbfdf94 --- Tracing command ls pid 1838 tid 100179 td 0xca3d6680 kdb_enter(c157cefa,c157cefa,c1584bdc,f39754f4,0,...) at kdb_enter+0x3a/frame 0xf39754a0 vpanic(c1584bdc,f39754f4,f39754f4,f3975580,c0c85f5b,...) at vpanic+0x132/frame 0xf39754d4 panic(c1584bdc,1878f,706,d3,c19dd2c8,...) at panic+0x1b/frame 0xf39754e8 propagate_priority(ca3d6680,0,c15847f0,2e5,c96f1800,...) at propagate_priority+0x2db/frame 0xf3975580 turnstile_wait(c96f1800,caece340,0,202,c180f27c,...) at turnstile_wait+0x80b/frame 0xf3975640 __mtx_lock_sleep(c8e40a30,ca3d6680,0,c1566c5f,67,...) at __mtx_lock_sleep+0x727/frame 0xf3975788 __mtx_lock_flags(c8e40a30,0,c1566c5f,67,0,...) at __mtx_lock_flags+0x225/frame 0xf39757e8 pfs_lookup(f39758d0,c177058e,c0cde7a8,cb99c5d4,f39758c8,...) at pfs_lookup+0x1d6/frame 0xf3975854 VOP_CACHEDLOOKUP_APV(c180f27c,f39758d0,f3975b44,0,0,...) at VOP_CACHEDLOOKUP_APV+0x21e/frame 0xf39758b0 vfs_cache_lookup(f3975990,c177052d,c95e92ec,80000,f3975938,...) at vfs_cache_lookup+0xe9/frame 0xf39758f4 VOP_LOOKUP_APV(c180f27c,f3975990,f3975b44,211,c1a09740,...) at VOP_LOOKUP_APV+0x21e/frame 0xf3975950 lookup(f3975ae8,c158ed16,124,eb,cbe02400,...) at lookup+0x53d/frame 0xf39759b8 namei(f3975ae8,0,4000104,0,28669db8,...) at namei+0x5df/frame 0xf3975a48 kern_statat(ca3d6680,200,ffffff9c,28669db8,0,...) at kern_statat+0xae/frame 0xf3975b80 sys_fstatat(ca3d6680,f3975ca8,c1574eec,4,c119f5d9,...) at sys_fstatat+0x49/frame 0xf3975c20 syscall(f3975ce8) at syscall+0x388/frame 0xf3975cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3975cdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- Tracing command ls pid 1830 tid 100228 td 0xca3d6340 sched_switch(ca3d6340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a3e680 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a3e6b8 sleepq_switch(ca3d6340,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3a3e6ec sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3a3e710 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3a3e758 __lockmgr_args(cb8a3304,80100,cb8a3370,0,0,0,c158f93c,9c2) at __lockmgr_args+0xfed/frame 0xf3a3e7d4 vop_stdlock(f3a3e878,c0bee139,c89a0178,c89a7950,c89a7950,...) at vop_stdlock+0x53/frame 0xf3a3e804 VOP_LOCK1_APV(c180f27c,f3a3e878,f3a3e8b4,f3a3e8b4,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3a3e860 _vn_lock(cb8a32d0,80100,c158f93c,9c2,ca3d6340,...) at _vn_lock+0xba/frame 0xf3a3e8a0 vget(cb8a32d0,80100,ca3d6340,81,c0b6bc50,...) at vget+0x122/frame 0xf3a3e8dc pfs_vncache_alloc(c95e92ec,f3a3e954,c8e40c00,186a0) at pfs_vncache_alloc+0xc6/frame 0xf3a3e900 pfs_root(c95e92ec,82000,f3a3e954,211,c1a09c68,...) at pfs_root+0x2d/frame 0xf3a3e918 lookup(f3a3eb50,c158ed16,124,eb,c181dacc,...) at lookup+0x7b4/frame 0xf3a3e980 namei(f3a3eb50,4,4,0,fc,...) at namei+0x5df/frame 0xf3a3ea10 vn_open_cred(f3a3eb50,f3a3ebdc,0,0,c95f3c00,c9701380) at vn_open_cred+0x1fd/frame 0xf3a3eae0 vn_open(f3a3eb50,f3a3ebdc,0,c9701380,28653000,...) at vn_open+0x3d/frame 0xf3a3eb08 kern_openat(ca3d6340,ffffff9c,28653000,0,120004,2864e000) at kern_openat+0x361/frame 0xf3a3ebfc sys_open(ca3d6340,f3a3eca8,c1574eec,4,c119f5d9,...) at sys_open+0x39/frame 0xf3a3ec20 syscall(f3a3ece8) at syscall+0x388/frame 0xf3a3ecdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a3ecdc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2822c74b, esp = 0xbfbfe0fc, ebp = 0xbfbfe114 --- Tracing command ls pid 1829 tid 100258 td 0xcbd8b9c0 sched_switch(cbd8b9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3aa9680 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3aa96b8 sleepq_switch(cbd8b9c0,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3aa96ec sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3aa9710 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3aa9758 __lockmgr_args(cb8a3304,80100,cb8a3370,0,0,0,c158f93c,9c2) at __lockmgr_args+0xfed/frame 0xf3aa97d4 vop_stdlock(f3aa9878,c0bee139,c89a0178,c89a7950,c89a7950,...) at vop_stdlock+0x53/frame 0xf3aa9804 VOP_LOCK1_APV(c180f27c,f3aa9878,f3aa98b4,f3aa98b4,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3aa9860 _vn_lock(cb8a32d0,80100,c158f93c,9c2,cbd8b9c0,...) at _vn_lock+0xba/frame 0xf3aa98a0 vget(cb8a32d0,80100,cbd8b9c0,81,c0b6bc50,...) at vget+0x122/frame 0xf3aa98dc pfs_vncache_alloc(c95e92ec,f3aa9954,c8e40c00,186a0) at pfs_vncache_alloc+0xc6/frame 0xf3aa9900 pfs_root(c95e92ec,82000,f3aa9954,211,c1a096e8,...) at pfs_root+0x2d/frame 0xf3aa9918 lookup(f3aa9b50,c158ed16,124,eb,c181dacc,...) at lookup+0x7b4/frame 0xf3aa9980 namei(f3aa9b50,4,4,0,fc,...) at namei+0x5df/frame 0xf3aa9a10 vn_open_cred(f3aa9b50,f3aa9bdc,0,0,c95f3c00,c9700cb0) at vn_open_cred+0x1fd/frame 0xf3aa9ae0 vn_open(f3aa9b50,f3aa9bdc,0,c9700cb0,28653000,...) at vn_open+0x3d/frame 0xf3aa9b08 kern_openat(cbd8b9c0,ffffff9c,28653000,0,120004,2864e000) at kern_openat+0x361/frame 0xf3aa9bfc sys_open(cbd8b9c0,f3aa9ca8,c1574eec,4,c119f5d9,...) at sys_open+0x39/frame 0xf3aa9c20 syscall(f3aa9ce8) at syscall+0x388/frame 0xf3aa9cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3aa9cdc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2822c74b, esp = 0xbfbfe0fc, ebp = 0xbfbfe114 --- Tracing command ls pid 1825 tid 100230 td 0xcbd6c9c0 sched_switch(cbd6c9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a44760 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a44798 sleepq_switch(cbd6c9c0,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3a447cc sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3a447f0 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3a44838 __lockmgr_args(cb8a3304,80100,cb8a3370,0,0,0,c158f93c,9c2) at __lockmgr_args+0xfed/frame 0xf3a448b4 vop_stdlock(f3a44958,c0bee139,c89a0178,c89a7950,c89a7950,...) at vop_stdlock+0x53/frame 0xf3a448e4 VOP_LOCK1_APV(c180f27c,f3a44958,f3a44994,f3a44994,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3a44940 _vn_lock(cb8a32d0,80100,c158f93c,9c2,cbd6c9c0,...) at _vn_lock+0xba/frame 0xf3a44980 vget(cb8a32d0,80100,cbd6c9c0,81,c0b6bc50,...) at vget+0x122/frame 0xf3a449bc pfs_vncache_alloc(c95e92ec,f3a44a34,c8e40c00,186a0) at pfs_vncache_alloc+0xc6/frame 0xf3a449e0 pfs_root(c95e92ec,82000,f3a44a34,211,c1a09638,...) at pfs_root+0x2d/frame 0xf3a449f8 lookup(f3a44b20,c158ed16,124,eb,c158e943,...) at lookup+0x7b4/frame 0xf3a44a60 namei(f3a44b20,0,4000104,0,bfbfd48b,...) at namei+0x5df/frame 0xf3a44af0 kern_readlinkat(cbd6c9c0,ffffff9c,bfbfd48b,0,bfbfd88c,0,400) at kern_readlinkat+0x6f/frame 0xf3a44bf8 sys_readlink(cbd6c9c0,f3a44ca8,c1574eec,4,c119f5d9,...) at sys_readlink+0x41/frame 0xf3a44c20 syscall(f3a44ce8) at syscall+0x388/frame 0xf3a44cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a44cdc --- syscall (58, FreeBSD ELF32, sys_readlink), eip = 0x2819fe87, esp = 0xbfbfd40c, ebp = 0xbfbfdca0 --- Tracing command ls pid 1798 tid 100239 td 0xcaece340 sched_switch(caece340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a658bc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3a658f4 sleepq_switch(caece340,0,c1583a69,29c,caece340,...) at sleepq_switch+0x127/frame 0xf3a65928 sleepq_timedwait(c19be8c1,0,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf3a6594c _sleep(c19be8c1,0,400,c1575203,b020c3d,...) at _sleep+0x362/frame 0xf3a659ac pause_sbt_pri(c1575203,400,b020c3d,0,0,...) at pause_sbt_pri+0x108/frame 0xf3a659e8 fail_point_eval_nontrivial(c1820938,f3a65a60,c157ab75,144,0,...) at fail_point_eval_nontrivial+0x2e4/frame 0xf3a65a40 _sx_slock(c1a93ef0,0,c157ab75,144,1000,...) at _sx_slock+0x16f/frame 0xf3a65a74 pfind(0,0,c1566c5f,67,0,...) at pfind+0x2e/frame 0xf3a65a90 pfs_readdir(f3a65bc0,c1770ce4,f3a65b10,200400,cba5e2d0,...) at pfs_readdir+0x152/frame 0xf3a65af4 VOP_READDIR_APV(c180f27c,f3a65bc0,c1591230,f9c,0,...) at VOP_READDIR_APV+0x21e/frame 0xf3a65b50 kern_getdirentries(caece340,5,2864d000,1000,f3a65c10,...) at kern_getdirentries+0x21b/frame 0xf3a65bec sys_getdirentries(caece340,f3a65ca8,c1574eec,4,c119f5d9,...) at sys_getdirentries+0x42/frame 0xf3a65c20 syscall(f3a65ce8) at syscall+0x388/frame 0xf3a65cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a65cdc --- syscall (196, FreeBSD ELF32, sys_getdirentries), eip = 0x28220cbb, esp = 0xbfbfe0c0, ebp = 0xbfbfe0f8 --- Tracing command ls pid 1796 tid 100261 td 0xcbe3a000 sched_switch(cbe3a000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3ab7730 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3ab7768 sleepq_switch(cbe3a000,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3ab779c sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3ab77c0 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3ab7808 __lockmgr_args(cb8a3304,82400,cb8a3370,0,0,0,c158ed16,211) at __lockmgr_args+0xfed/frame 0xf3ab7884 vop_stdlock(f3ab7928,f3ab7900,c181dacc,cbe3a000,f3ab7928,...) at vop_stdlock+0x53/frame 0xf3ab78b4 VOP_LOCK1_APV(c180f27c,f3ab7928,c8f7ba10,c915d1e0,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3ab7910 _vn_lock(cb8a32d0,82400,c158ed16,211,c1a09fd8,...) at _vn_lock+0xba/frame 0xf3ab7950 lookup(f3ab7ae8,c158ed16,124,eb,cb9e4920,...) at lookup+0xf1/frame 0xf3ab79b8 namei(f3ab7ae8,0,4000104,0,2865a7b8,...) at namei+0x5df/frame 0xf3ab7a48 kern_statat(cbe3a000,200,ffffff9c,2865a7b8,0,...) at kern_statat+0xae/frame 0xf3ab7b80 sys_fstatat(cbe3a000,f3ab7ca8,c1574eec,4,f3ab7c54,...) at sys_fstatat+0x49/frame 0xf3ab7c20 syscall(f3ab7ce8) at syscall+0x388/frame 0xf3ab7cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ab7cdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- Tracing command ls pid 1793 tid 100236 td 0xcbdb0340 sched_switch(cbdb0340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a5c730 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a5c768 sleepq_switch(cbdb0340,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3a5c79c sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3a5c7c0 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3a5c808 __lockmgr_args(cb8a3304,82400,cb8a3370,0,0,0,c158ed16,211) at __lockmgr_args+0xfed/frame 0xf3a5c884 vop_stdlock(f3a5c928,f3a5c900,c181dacc,cbdb0340,f3a5c928,...) at vop_stdlock+0x53/frame 0xf3a5c8b4 VOP_LOCK1_APV(c180f27c,f3a5c928,c8f7ba10,c915d1e0,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3a5c910 _vn_lock(cb8a32d0,82400,c158ed16,211,c1a09798,...) at _vn_lock+0xba/frame 0xf3a5c950 lookup(f3a5cae8,c158ed16,124,eb,cb82d650,...) at lookup+0xf1/frame 0xf3a5c9b8 namei(f3a5cae8,0,4000104,0,28658778,...) at namei+0x5df/frame 0xf3a5ca48 kern_statat(cbdb0340,200,ffffff9c,28658778,0,...) at kern_statat+0xae/frame 0xf3a5cb80 sys_fstatat(cbdb0340,f3a5cca8,c1574eec,4,f3a5cc54,...) at sys_fstatat+0x49/frame 0xf3a5cc20 syscall(f3a5cce8) at syscall+0x388/frame 0xf3a5ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a5ccdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- Tracing command ls pid 1790 tid 100095 td 0xc973c000 sched_switch(c973c000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf448f8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf44930 sleepq_switch(c973c000,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xedf44964 sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xedf44988 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xedf449d0 __lockmgr_args(cb8a3304,200400,cb8a3370,0,0,0,c1591450,53c) at __lockmgr_args+0xfed/frame 0xedf44a4c vop_stdlock(edf44af0,c1a0a450,c1a0a440,c1a0a440,8,...) at vop_stdlock+0x53/frame 0xedf44a7c VOP_LOCK1_APV(c180f27c,edf44af0,2,c97444b8,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xedf44ad8 _vn_lock(cb8a32d0,200400,c1591450,53c,ffffffff,...) at _vn_lock+0xba/frame 0xedf44b18 vn_statfile(c97807a8,edf44bb8,c95f3c00,c973c000,0,...) at vn_statfile+0x35/frame 0xedf44b58 kern_fstat(c973c000,5,edf44bb8,c19ccc90,c19ccc80,...) at kern_fstat+0xde/frame 0xedf44ba0 sys_fstat(c973c000,edf44ca8,c1574eec,4,edf44c54,...) at sys_fstat+0x27/frame 0xedf44c20 syscall(edf44ce8) at syscall+0x388/frame 0xedf44cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf44cdc --- syscall (189, FreeBSD ELF32, sys_fstat), eip = 0x28232b2f, esp = 0xbfbfe084, ebp = 0xbfbfe114 --- Tracing command ls pid 1789 tid 100163 td 0xca2ab680 sched_switch(ca2ab680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf393a730 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf393a768 sleepq_switch(ca2ab680,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf393a79c sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf393a7c0 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf393a808 __lockmgr_args(cb8a3304,82400,cb8a3370,0,0,0,c158ed16,211) at __lockmgr_args+0xfed/frame 0xf393a884 vop_stdlock(f393a928,f393a900,c181dacc,ca2ab680,f393a928,...) at vop_stdlock+0x53/frame 0xf393a8b4 VOP_LOCK1_APV(c180f27c,f393a928,c8f7ba10,c915d1e0,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf393a910 _vn_lock(cb8a32d0,82400,c158ed16,211,c1a09b08,...) at _vn_lock+0xba/frame 0xf393a950 lookup(f393aae8,c158ed16,124,eb,c181dbf4,...) at lookup+0xf1/frame 0xf393a9b8 namei(f393aae8,0,4000104,0,28669cf8,...) at namei+0x5df/frame 0xf393aa48 kern_statat(ca2ab680,200,ffffff9c,28669cf8,0,...) at kern_statat+0xae/frame 0xf393ab80 sys_fstatat(ca2ab680,f393aca8,c1574eec,4,f393ac54,...) at sys_fstatat+0x49/frame 0xf393ac20 syscall(f393ace8) at syscall+0x388/frame 0xf393acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf393acdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- Tracing command ls pid 1787 tid 100267 td 0xcbe71000 sched_switch(cbe71000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3ac95e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3ac9620 sleepq_switch(cbe71000,0,c1583a69,279,cba5e304,...) at sleepq_switch+0x127/frame 0xf3ac9654 sleepq_wait(cba5e304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3ac9678 sleeplk(cba5e370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3ac96c0 __lockmgr_args(cba5e304,80400,cba5e370,0,0,0,c158dfb9,2b0) at __lockmgr_args+0xfed/frame 0xf3ac973c vop_stdlock(f3ac97e0,c18295d8,0,86000,cb8a32d0,...) at vop_stdlock+0x53/frame 0xf3ac976c VOP_LOCK1_APV(c180f27c,f3ac97e0,c158f93c,9c2,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3ac97c8 _vn_lock(cba5e2d0,80400,c158dfb9,2b0,c11c791b,...) at _vn_lock+0xba/frame 0xf3ac9808 cache_lookup(cba5e2d0,f3ac9b98,f3ac9bac,0,0,...) at cache_lookup+0x82c/frame 0xf3ac9878 vfs_cache_lookup(f3ac9958,c177052d,c95e92ec,80000,f3ac9900,...) at vfs_cache_lookup+0xb5/frame 0xf3ac98bc VOP_LOOKUP_APV(c180f27c,f3ac9958,f3ac9bac,211,c1a0a240,...) at VOP_LOOKUP_APV+0x21e/frame 0xf3ac9918 lookup(f3ac9b50,c158ed16,124,eb,f3ac99a4,...) at lookup+0x53d/frame 0xf3ac9980 namei(f3ac9b50,4,4,0,fc,...) at namei+0x5df/frame 0xf3ac9a10 vn_open_cred(f3ac9b50,f3ac9bdc,0,0,c95f3c00,c9745700) at vn_open_cred+0x1fd/frame 0xf3ac9ae0 vn_open(f3ac9b50,f3ac9bdc,0,c9745700,28240ccc,...) at vn_open+0x3d/frame 0xf3ac9b08 kern_openat(cbe71000,ffffff9c,28240ccc,0,120000,0) at kern_openat+0x361/frame 0xf3ac9bfc sys_open(cbe71000,f3ac9ca8,c1574eec,4,c119f5d9,...) at sys_open+0x39/frame 0xf3ac9c20 syscall(f3ac9ce8) at syscall+0x388/frame 0xf3ac9cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3ac9cdc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2822c74b, esp = 0xbfbfe0fc, ebp = 0xbfbfe18c --- Tracing command ls pid 1771 tid 100225 td 0xcaecf000 sched_switch(caecf000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a31730 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a31768 sleepq_switch(caecf000,0,c1583a69,279,cb8a3304,...) at sleepq_switch+0x127/frame 0xf3a3179c sleepq_wait(cb8a3304,60,c156677b,4,0,...) at sleepq_wait+0x48/frame 0xf3a317c0 sleeplk(cb8a3370,c156677b,60,33,0,...) at sleeplk+0x1a6/frame 0xf3a31808 __lockmgr_args(cb8a3304,82400,cb8a3370,0,0,0,c158ed16,211) at __lockmgr_args+0xfed/frame 0xf3a31884 vop_stdlock(f3a31928,f3a31900,c181dacc,caecf000,f3a31928,...) at vop_stdlock+0x53/frame 0xf3a318b4 VOP_LOCK1_APV(c180f27c,f3a31928,c8f7ba10,c915d1e0,c185e36c,...) at VOP_LOCK1_APV+0x202/frame 0xf3a31910 _vn_lock(cb8a32d0,82400,c158ed16,211,c1a09588,...) at _vn_lock+0xba/frame 0xf3a31950 lookup(f3a31ae8,c158ed16,124,eb,cba5e650,...) at lookup+0xf1/frame 0xf3a319b8 namei(f3a31ae8,0,4000104,0,28659678,...) at namei+0x5df/frame 0xf3a31a48 kern_statat(caecf000,200,ffffff9c,28659678,0,...) at kern_statat+0xae/frame 0xf3a31b80 sys_fstatat(caecf000,f3a31ca8,c1574eec,4,c119f5d9,...) at sys_fstatat+0x49/frame 0xf3a31c20 syscall(f3a31ce8) at syscall+0x388/frame 0xf3a31cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a31cdc --- syscall (493, FreeBSD ELF32, sys_fstatat), eip = 0x2816f367, esp = 0xbfbfe07c, ebp = 0xbfbfe114 --- Tracing command sh pid 1073 tid 100223 td 0xcbdb1340 sched_switch(cbdb1340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a2a968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a2a9a0 sleepq_switch(cbdb1340,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf3a2a9d4 sleepq_catch_signals(c1583a69,16f,14,2c1,cbdb1340,...) at sleepq_catch_signals+0x3b7/frame 0xf3a2aa10 sleepq_wait_sig(caeeb358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a2aa34 _sleep(caeeb358,caeeb400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a2aa94 kern_wait6(cbdb1340,7,0,0,f3a2ab2c,...) at kern_wait6+0x47e/frame 0xf3a2ab00 sys_wait4(cbdb1340,f3a2aca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf3a2ac20 syscall(f3a2ace8) at syscall+0x388/frame 0xf3a2acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a2acdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1072 tid 100222 td 0xcbdb1680 sched_switch(cbdb1680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a26968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a269a0 sleepq_switch(cbdb1680,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a269d4 sleepq_catch_signals(c1583a69,16f,14,12,cbdb1680,...) at sleepq_catch_signals+0x3b7/frame 0xf3a26a10 sleepq_wait_sig(caeeb6b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a26a34 _sleep(caeeb6b0,caeeb758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a26a94 kern_wait6(cbdb1680,7,0,0,f3a26b2c,...) at kern_wait6+0x47e/frame 0xf3a26b00 sys_wait4(cbdb1680,f3a26ca8,c1574eec,4,f3a26c54,...) at sys_wait4+0x8b/frame 0xf3a26c20 syscall(f3a26ce8) at syscall+0x388/frame 0xf3a26cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a26cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1071 tid 100221 td 0xcbdb19c0 sched_switch(cbdb19c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a22968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a229a0 sleepq_switch(cbdb19c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf3a229d4 sleepq_catch_signals(c1583a69,16f,14,216,cbdb19c0,...) at sleepq_catch_signals+0x3b7/frame 0xf3a22a10 sleepq_wait_sig(caeeba08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a22a34 _sleep(caeeba08,caeebab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a22a94 kern_wait6(cbdb19c0,7,0,0,f3a22b2c,...) at kern_wait6+0x47e/frame 0xf3a22b00 sys_wait4(cbdb19c0,f3a22ca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf3a22c20 syscall(f3a22ce8) at syscall+0x388/frame 0xf3a22cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a22cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1070 tid 100220 td 0xcbd6d680 sched_switch(cbd6d680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a1e9e4 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3a1ea1c sleepq_switch(cbd6d680,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf3a1ea50 sleepq_catch_signals(c1583a69,16f,14,c9,cbd6d680,...) at sleepq_catch_signals+0x3b7/frame 0xf3a1ea8c sleepq_wait_sig(c97b7330,5c,c1587368,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a1eab0 _sleep(c97b7330,c97b74b0,15c,c1587368,0,...) at _sleep+0x37e/frame 0xf3a1eb10 pipe_read(c9780738,f3a1ebf0,c95f3c00,0,cbd6d680,...) at pipe_read+0x2ca/frame 0xf3a1eb54 dofileread(c9780738,f3a1ebf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3a1eb88 kern_readv(cbd6d680,3,f3a1ebf0,cbd9d000,bfbfe438,...) at kern_readv+0x93/frame 0xf3a1ebd0 sys_read(cbd6d680,f3a1eca8,c1574eec,4,c181dbf4,...) at sys_read+0x59/frame 0xf3a1ec20 syscall(f3a1ece8) at syscall+0x388/frame 0xf3a1ecdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a1ecdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28204553, esp = 0xbfbfe39c, ebp = 0xbfbfe3b0 --- Tracing command sh pid 1069 tid 100219 td 0xcbd6d9c0 sched_switch(cbd6d9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a1a968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a1a9a0 sleepq_switch(cbd6d9c0,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a1a9d4 sleepq_catch_signals(c1583a69,16f,14,1a1,cbd6d9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf3a1aa10 sleepq_wait_sig(cbd9d358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a1aa34 _sleep(cbd9d358,cbd9d400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a1aa94 kern_wait6(cbd6d9c0,7,0,0,f3a1ab2c,...) at kern_wait6+0x47e/frame 0xf3a1ab00 sys_wait4(cbd6d9c0,f3a1aca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf3a1ac20 syscall(f3a1ace8) at syscall+0x388/frame 0xf3a1acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a1acdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1068 tid 100218 td 0xcbd89000 sched_switch(cbd89000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a16968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a169a0 sleepq_switch(cbd89000,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a169d4 sleepq_catch_signals(c1583a69,16f,14,132,cbd89000,...) at sleepq_catch_signals+0x3b7/frame 0xf3a16a10 sleepq_wait_sig(cbd9d6b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a16a34 _sleep(cbd9d6b0,cbd9d758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a16a94 kern_wait6(cbd89000,7,0,0,f3a16b2c,...) at kern_wait6+0x47e/frame 0xf3a16b00 sys_wait4(cbd89000,f3a16ca8,c1574eec,4,c181dbf4,...) at sys_wait4+0x8b/frame 0xf3a16c20 syscall(f3a16ce8) at syscall+0x388/frame 0xf3a16cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a16cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1066 tid 100217 td 0xcbd89340 sched_switch(cbd89340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a12968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a129a0 sleepq_switch(cbd89340,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf3a129d4 sleepq_catch_signals(c1583a69,16f,14,276,cbd89340,...) at sleepq_catch_signals+0x3b7/frame 0xf3a12a10 sleepq_wait_sig(cbd9da08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a12a34 _sleep(cbd9da08,cbd9dab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a12a94 kern_wait6(cbd89340,7,0,0,f3a12b2c,...) at kern_wait6+0x47e/frame 0xf3a12b00 sys_wait4(cbd89340,f3a12ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf3a12c20 syscall(f3a12ce8) at syscall+0x388/frame 0xf3a12cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a12cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1065 tid 100216 td 0xcbd89680 sched_switch(cbd89680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a0e8f0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a0e928 sleepq_switch(cbd89680,0,c1583a69,279,2711,...) at sleepq_switch+0x127/frame 0xf3a0e95c sleepq_wait(c1a93ef0,0,c157ab5c,3,0,...) at sleepq_wait+0x48/frame 0xf3a0e980 _sx_xlock_hard(c1a93ef0,cbd89680,0,c1574f40,348,...) at _sx_xlock_hard+0x513/frame 0xf3a0ea04 _sx_xlock(c1a93ef0,0,c1574f40,348,cbd9e0b8,...) at _sx_xlock+0xfe/frame 0xf3a0ea38 proc_reap(cbd89680,cbdbe358,f3a0eb2c,30,cbd89680,...) at proc_reap+0x1a2/frame 0xf3a0ea68 proc_to_reap(7,0,0,f3a0eb2c,30,...) at proc_to_reap+0x339/frame 0xf3a0ea94 kern_wait6(cbd89680,7,0,0,f3a0eb2c,...) at kern_wait6+0x232/frame 0xf3a0eb00 sys_wait4(cbd89680,f3a0eca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf3a0ec20 syscall(f3a0ece8) at syscall+0x388/frame 0xf3a0ecdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a0ecdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1064 tid 100215 td 0xcbd899c0 sched_switch(cbd899c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a0b968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a0b9a0 sleepq_switch(cbd899c0,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf3a0b9d4 sleepq_catch_signals(c1583a69,16f,14,231,cbd899c0,...) at sleepq_catch_signals+0x3b7/frame 0xf3a0ba10 sleepq_wait_sig(cbd9e358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a0ba34 _sleep(cbd9e358,cbd9e400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a0ba94 kern_wait6(cbd899c0,7,0,0,f3a0bb2c,...) at kern_wait6+0x47e/frame 0xf3a0bb00 sys_wait4(cbd899c0,f3a0bca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf3a0bc20 syscall(f3a0bce8) at syscall+0x388/frame 0xf3a0bcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a0bcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1056 tid 100213 td 0xcbd8a340 sched_switch(cbd8a340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a04968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a049a0 sleepq_switch(cbd8a340,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a049d4 sleepq_catch_signals(c1583a69,16f,14,2a6,cbd8a340,...) at sleepq_catch_signals+0x3b7/frame 0xf3a04a10 sleepq_wait_sig(cbd9ea08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a04a34 _sleep(cbd9ea08,cbd9eab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a04a94 kern_wait6(cbd8a340,7,0,0,f3a04b2c,...) at kern_wait6+0x47e/frame 0xf3a04b00 sys_wait4(cbd8a340,f3a04ca8,c1574eec,4,f3a04c54,...) at sys_wait4+0x8b/frame 0xf3a04c20 syscall(f3a04ce8) at syscall+0x388/frame 0xf3a04cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a04cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1055 tid 100212 td 0xcbd8a680 sched_switch(cbd8a680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3a00968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3a009a0 sleepq_switch(cbd8a680,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3a009d4 sleepq_catch_signals(c1583a69,16f,14,2d0,cbd8a680,...) at sleepq_catch_signals+0x3b7/frame 0xf3a00a10 sleepq_wait_sig(cbd9f000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3a00a34 _sleep(cbd9f000,cbd9f0a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3a00a94 kern_wait6(cbd8a680,7,0,0,f3a00b2c,...) at kern_wait6+0x47e/frame 0xf3a00b00 sys_wait4(cbd8a680,f3a00ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf3a00c20 syscall(f3a00ce8) at syscall+0x388/frame 0xf3a00cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3a00cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1054 tid 100211 td 0xcbd8a9c0 sched_switch(cbd8a9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39fd968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39fd9a0 sleepq_switch(cbd8a9c0,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf39fd9d4 sleepq_catch_signals(c1583a69,16f,14,201,cbd8a9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf39fda10 sleepq_wait_sig(cbd9f358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39fda34 _sleep(cbd9f358,cbd9f400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39fda94 kern_wait6(cbd8a9c0,7,0,0,f39fdb2c,...) at kern_wait6+0x47e/frame 0xf39fdb00 sys_wait4(cbd8a9c0,f39fdca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf39fdc20 syscall(f39fdce8) at syscall+0x388/frame 0xf39fdcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39fdcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1053 tid 100091 td 0xc96f49c0 sched_switch(c96f49c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf2c968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf2c9a0 sleepq_switch(c96f49c0,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xedf2c9d4 sleepq_catch_signals(c1583a69,16f,14,261,c96f49c0,...) at sleepq_catch_signals+0x3b7/frame 0xedf2ca10 sleepq_wait_sig(c96e9358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xedf2ca34 _sleep(c96e9358,c96e9400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xedf2ca94 kern_wait6(c96f49c0,7,0,0,edf2cb2c,...) at kern_wait6+0x47e/frame 0xedf2cb00 sys_wait4(c96f49c0,edf2cca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xedf2cc20 syscall(edf2cce8) at syscall+0x388/frame 0xedf2ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf2ccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1052 tid 100181 td 0xcaed1000 sched_switch(caed1000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3985968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39859a0 sleepq_switch(caed1000,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf39859d4 sleepq_catch_signals(c1583a69,16f,14,1c2,caed1000,...) at sleepq_catch_signals+0x3b7/frame 0xf3985a10 sleepq_wait_sig(c9aa26b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3985a34 _sleep(c9aa26b0,c9aa2758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf3985a94 kern_wait6(caed1000,7,0,0,f3985b2c,...) at kern_wait6+0x47e/frame 0xf3985b00 sys_wait4(caed1000,f3985ca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf3985c20 syscall(f3985ce8) at syscall+0x388/frame 0xf3985cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3985cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1050 tid 100183 td 0xcaed09c0 sched_switch(caed09c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf398b968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf398b9a0 sleepq_switch(caed09c0,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf398b9d4 sleepq_catch_signals(c1583a69,16f,14,171,caed09c0,...) at sleepq_catch_signals+0x3b7/frame 0xf398ba10 sleepq_wait_sig(c9aa2358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf398ba34 _sleep(c9aa2358,c9aa2400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf398ba94 kern_wait6(caed09c0,7,0,0,f398bb2c,...) at kern_wait6+0x47e/frame 0xf398bb00 sys_wait4(caed09c0,f398bca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf398bc20 syscall(f398bce8) at syscall+0x388/frame 0xf398bcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf398bcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1049 tid 100184 td 0xcaed0680 sched_switch(caed0680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf398e968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf398e9a0 sleepq_switch(caed0680,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf398e9d4 sleepq_catch_signals(c1583a69,16f,14,60,caed0680,...) at sleepq_catch_signals+0x3b7/frame 0xf398ea10 sleepq_wait_sig(c9aa2000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf398ea34 _sleep(c9aa2000,c9aa20a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf398ea94 kern_wait6(caed0680,7,0,0,f398eb2c,...) at kern_wait6+0x47e/frame 0xf398eb00 sys_wait4(caed0680,f398eca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf398ec20 syscall(f398ece8) at syscall+0x388/frame 0xf398ecdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf398ecdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1048 tid 100188 td 0xca2ab000 sched_switch(ca2ab000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf399a968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf399a9a0 sleepq_switch(ca2ab000,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf399a9d4 sleepq_catch_signals(c1583a69,16f,14,282,ca2ab000,...) at sleepq_catch_signals+0x3b7/frame 0xf399aa10 sleepq_wait_sig(ca2d66b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf399aa34 _sleep(ca2d66b0,ca2d6758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf399aa94 kern_wait6(ca2ab000,7,0,0,f399ab2c,...) at kern_wait6+0x47e/frame 0xf399ab00 sys_wait4(ca2ab000,f399aca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf399ac20 syscall(f399ace8) at syscall+0x388/frame 0xf399acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf399acdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1047 tid 100189 td 0xca2aa9c0 sched_switch(ca2aa9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf399d968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf399d9a0 sleepq_switch(ca2aa9c0,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf399d9d4 sleepq_catch_signals(c1583a69,16f,14,b1,ca2aa9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf399da10 sleepq_wait_sig(ca2d6358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf399da34 _sleep(ca2d6358,ca2d6400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf399da94 kern_wait6(ca2aa9c0,7,0,0,f399db2c,...) at kern_wait6+0x47e/frame 0xf399db00 sys_wait4(ca2aa9c0,f399dca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf399dc20 syscall(f399dce8) at syscall+0x388/frame 0xf399dcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf399dcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 1046 tid 100190 td 0xca2aa680 sched_switch(ca2aa680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39a0968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39a09a0 sleepq_switch(ca2aa680,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf39a09d4 sleepq_catch_signals(c1583a69,16f,14,120,ca2aa680,...) at sleepq_catch_signals+0x3b7/frame 0xf39a0a10 sleepq_wait_sig(ca2d6000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39a0a34 _sleep(ca2d6000,ca2d60a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39a0a94 kern_wait6(ca2aa680,7,0,0,f39a0b2c,...) at kern_wait6+0x47e/frame 0xf39a0b00 sys_wait4(ca2aa680,f39a0ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39a0c20 syscall(f39a0ce8) at syscall+0x388/frame 0xf39a0cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39a0cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1044 tid 100161 td 0xca2ab9c0 sched_switch(ca2ab9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf539e4 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedf53a1c sleepq_switch(ca2ab9c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xedf53a50 sleepq_catch_signals(c1583a69,16f,14,72,ca2ab9c0,...) at sleepq_catch_signals+0x3b7/frame 0xedf53a8c sleepq_wait_sig(c9784660,5c,c1587368,100,0,...) at sleepq_wait_sig+0x14/frame 0xedf53ab0 _sleep(c9784660,c97847e0,15c,c1587368,0,...) at _sleep+0x37e/frame 0xedf53b10 pipe_read(c97585b0,edf53bf0,c95f3c00,0,ca2ab9c0,...) at pipe_read+0x2ca/frame 0xedf53b54 dofileread(c97585b0,edf53bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xedf53b88 kern_readv(ca2ab9c0,3,edf53bf0,ca2d7000,bfbfe438,...) at kern_readv+0x93/frame 0xedf53bd0 sys_read(ca2ab9c0,edf53ca8,c1574eec,4,edf53c54,...) at sys_read+0x59/frame 0xedf53c20 syscall(edf53ce8) at syscall+0x388/frame 0xedf53cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf53cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28204553, esp = 0xbfbfe39c, ebp = 0xbfbfe3b0 --- Tracing command sh pid 1042 tid 100177 td 0xca3c7340 sched_switch(ca3c7340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf396f968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf396f9a0 sleepq_switch(ca3c7340,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf396f9d4 sleepq_catch_signals(c1583a69,16f,14,186,ca3c7340,...) at sleepq_catch_signals+0x3b7/frame 0xf396fa10 sleepq_wait_sig(c9378a08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf396fa34 _sleep(c9378a08,c9378ab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf396fa94 kern_wait6(ca3c7340,7,0,0,f396fb2c,...) at kern_wait6+0x47e/frame 0xf396fb00 sys_wait4(ca3c7340,f396fca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf396fc20 syscall(f396fce8) at syscall+0x388/frame 0xf396fcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf396fcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1041 tid 100210 td 0xcbd8b000 sched_switch(cbd8b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39fa968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39fa9a0 sleepq_switch(cbd8b000,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf39fa9d4 sleepq_catch_signals(c1583a69,16f,14,180,cbd8b000,...) at sleepq_catch_signals+0x3b7/frame 0xf39faa10 sleepq_wait_sig(caee8000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39faa34 _sleep(caee8000,caee80a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39faa94 kern_wait6(cbd8b000,7,0,0,f39fab2c,...) at kern_wait6+0x47e/frame 0xf39fab00 sys_wait4(cbd8b000,f39faca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf39fac20 syscall(f39face8) at syscall+0x388/frame 0xf39facdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39facdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1040 tid 100209 td 0xcbd8b340 sched_switch(cbd8b340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39f7968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39f79a0 sleepq_switch(cbd8b340,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39f79d4 sleepq_catch_signals(c1583a69,16f,14,291,cbd8b340,...) at sleepq_catch_signals+0x3b7/frame 0xf39f7a10 sleepq_wait_sig(caee8358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39f7a34 _sleep(caee8358,caee8400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39f7a94 kern_wait6(cbd8b340,7,0,0,f39f7b2c,...) at kern_wait6+0x47e/frame 0xf39f7b00 sys_wait4(cbd8b340,f39f7ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39f7c20 syscall(f39f7ce8) at syscall+0x388/frame 0xf39f7cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39f7cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1039 tid 100208 td 0xcbd8b680 sched_switch(cbd8b680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39f4968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39f49a0 sleepq_switch(cbd8b680,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39f49d4 sleepq_catch_signals(c1583a69,16f,14,a2,cbd8b680,...) at sleepq_catch_signals+0x3b7/frame 0xf39f4a10 sleepq_wait_sig(caee86b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39f4a34 _sleep(caee86b0,caee8758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39f4a94 kern_wait6(cbd8b680,7,0,0,f39f4b2c,...) at kern_wait6+0x47e/frame 0xf39f4b00 sys_wait4(cbd8b680,f39f4ca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf39f4c20 syscall(f39f4ce8) at syscall+0x388/frame 0xf39f4cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39f4cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1038 tid 100207 td 0xca2ac680 sched_switch(ca2ac680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39f1968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39f19a0 sleepq_switch(ca2ac680,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf39f19d4 sleepq_catch_signals(c1583a69,16f,14,186,ca2ac680,...) at sleepq_catch_signals+0x3b7/frame 0xf39f1a10 sleepq_wait_sig(caee8a08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39f1a34 _sleep(caee8a08,caee8ab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39f1a94 kern_wait6(ca2ac680,7,0,0,f39f1b2c,...) at kern_wait6+0x47e/frame 0xf39f1b00 sys_wait4(ca2ac680,f39f1ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39f1c20 syscall(f39f1ce8) at syscall+0x388/frame 0xf39f1cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39f1cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1037 tid 100206 td 0xca2ac9c0 sched_switch(ca2ac9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39ee968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39ee9a0 sleepq_switch(ca2ac9c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39ee9d4 sleepq_catch_signals(c1583a69,16f,14,1b0,ca2ac9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf39eea10 sleepq_wait_sig(caee9000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39eea34 _sleep(caee9000,caee90a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39eea94 kern_wait6(ca2ac9c0,7,0,0,f39eeb2c,...) at kern_wait6+0x47e/frame 0xf39eeb00 sys_wait4(ca2ac9c0,f39eeca8,c1574eec,4,f39eec54,...) at sys_wait4+0x8b/frame 0xf39eec20 syscall(f39eece8) at syscall+0x388/frame 0xf39eecdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39eecdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1036 tid 100205 td 0xca3c5000 sched_switch(ca3c5000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39eb968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39eb9a0 sleepq_switch(ca3c5000,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39eb9d4 sleepq_catch_signals(c1583a69,16f,14,261,ca3c5000,...) at sleepq_catch_signals+0x3b7/frame 0xf39eba10 sleepq_wait_sig(caee9358,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39eba34 _sleep(caee9358,caee9400,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39eba94 kern_wait6(ca3c5000,7,0,0,f39ebb2c,...) at kern_wait6+0x47e/frame 0xf39ebb00 sys_wait4(ca3c5000,f39ebca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39ebc20 syscall(f39ebce8) at syscall+0x388/frame 0xf39ebcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39ebcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1035 tid 100204 td 0xca3c5340 sched_switch(ca3c5340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39c7968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39c79a0 sleepq_switch(ca3c5340,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf39c79d4 sleepq_catch_signals(c1583a69,16f,14,72,ca3c5340,...) at sleepq_catch_signals+0x3b7/frame 0xf39c7a10 sleepq_wait_sig(caee96b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39c7a34 _sleep(caee96b0,caee9758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39c7a94 kern_wait6(ca3c5340,7,0,0,f39c7b2c,...) at kern_wait6+0x47e/frame 0xf39c7b00 sys_wait4(ca3c5340,f39c7ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39c7c20 syscall(f39c7ce8) at syscall+0x388/frame 0xf39c7cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39c7cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe4d0, ebp = 0xbfbfe4e8 --- Tracing command sh pid 1034 tid 100192 td 0xca2aa000 sched_switch(ca2aa000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39a6968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39a69a0 sleepq_switch(ca2aa000,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf39a69d4 sleepq_catch_signals(c1583a69,16f,14,2b2,ca2aa000,...) at sleepq_catch_signals+0x3b7/frame 0xf39a6a10 sleepq_wait_sig(ca2d56b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39a6a34 _sleep(ca2d56b0,ca2d5758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39a6a94 kern_wait6(ca2aa000,7,0,0,f39a6b2c,...) at kern_wait6+0x47e/frame 0xf39a6b00 sys_wait4(ca2aa000,f39a6ca8,c1574eec,4,0,...) at sys_wait4+0x8b/frame 0xf39a6c20 syscall(f39a6ce8) at syscall+0x388/frame 0xf39a6cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39a6cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe590, ebp = 0xbfbfe5a8 --- Tracing command sh pid 1033 tid 100168 td 0xca189000 sched_switch(ca189000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf394c968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf394c9a0 sleepq_switch(ca189000,0,c1583a69,1b8,c19ccc90,...) at sleepq_switch+0x127/frame 0xf394c9d4 sleepq_catch_signals(c1583a69,16f,14,180,ca189000,...) at sleepq_catch_signals+0x3b7/frame 0xf394ca10 sleepq_wait_sig(c9a98000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf394ca34 _sleep(c9a98000,c9a980a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf394ca94 kern_wait6(ca189000,7,0,0,f394cb2c,...) at kern_wait6+0x47e/frame 0xf394cb00 sys_wait4(ca189000,f394cca8,c1574eec,4,c181dbf4,...) at sys_wait4+0x8b/frame 0xf394cc20 syscall(f394cce8) at syscall+0x388/frame 0xf394ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf394ccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe5e8, ebp = 0xbfbfe600 --- Tracing command sh pid 978 tid 100186 td 0xca3c6340 sched_switch(ca3c6340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3994a84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf3994abc sleepq_switch(ca3c6340,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf3994af0 sleepq_catch_signals(c1583a69,16f,14,219,ca3c6340,...) at sleepq_catch_signals+0x3b7/frame 0xf3994b2c sleepq_wait_sig(caeea714,74,c120768f,100,0,...) at sleepq_wait_sig+0x14/frame 0xf3994b50 _sleep(caeea714,caeea758,174,c120768f,0,...) at _sleep+0x37e/frame 0xf3994bb0 kern_sigsuspend(ca3c6340,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0xf3994bf0 sys_sigsuspend(ca3c6340,f3994ca8,c1574eec,4,c119f5d9,...) at sys_sigsuspend+0x55/frame 0xf3994c20 syscall(f3994ce8) at syscall+0x388/frame 0xf3994cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3994cdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x28195343, esp = 0xbfbfe89c, ebp = 0xbfbfe8a8 --- Tracing command sh pid 960 tid 100202 td 0xca3c5680 sched_switch(ca3c5680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39c1968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39c19a0 sleepq_switch(ca3c5680,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xf39c19d4 sleepq_catch_signals(c1583a69,16f,14,1b6,ca3c5680,...) at sleepq_catch_signals+0x3b7/frame 0xf39c1a10 sleepq_wait_sig(caee9a08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39c1a34 _sleep(caee9a08,caee9ab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39c1a94 kern_wait6(ca3c5680,7,0,0,f39c1b2c,...) at kern_wait6+0x47e/frame 0xf39c1b00 sys_wait4(ca3c5680,f39c1ca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf39c1c20 syscall(f39c1ce8) at syscall+0x388/frame 0xf39c1cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39c1cdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28195303, esp = 0xbfbfe840, ebp = 0xbfbfe858 --- Tracing command bash pid 954 tid 100201 td 0xca3c59c0 sched_switch(ca3c59c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39be968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39be9a0 sleepq_switch(ca3c59c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39be9d4 sleepq_catch_signals(c1583a69,16f,14,1e0,ca3c59c0,...) at sleepq_catch_signals+0x3b7/frame 0xf39bea10 sleepq_wait_sig(caeea000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39bea34 _sleep(caeea000,caeea0a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf39bea94 kern_wait6(ca3c59c0,7,0,0,f39beb2c,...) at kern_wait6+0x47e/frame 0xf39beb00 sys_wait4(ca3c59c0,f39beca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf39bec20 syscall(f39bece8) at syscall+0x388/frame 0xf39becdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39becdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28262303, esp = 0xbfbfe814, ebp = 0xbfbfe82c --- Tracing command csh pid 952 tid 100200 td 0xca3c6000 sched_switch(ca3c6000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39bba84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf39bbabc sleepq_switch(ca3c6000,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf39bbaf0 sleepq_catch_signals(c1583a69,16f,14,5d,ca3c6000,...) at sleepq_catch_signals+0x3b7/frame 0xf39bbb2c sleepq_wait_sig(caeea3bc,74,c120768f,100,0,...) at sleepq_wait_sig+0x14/frame 0xf39bbb50 _sleep(caeea3bc,caeea400,174,c120768f,0,...) at _sleep+0x37e/frame 0xf39bbbb0 kern_sigsuspend(ca3c6000,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0xf39bbbf0 sys_sigsuspend(ca3c6000,f39bbca8,c1574eec,4,0,...) at sys_sigsuspend+0x55/frame 0xf39bbc20 syscall(f39bbce8) at syscall+0x388/frame 0xf39bbcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf39bbcdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2820c343, esp = 0xbfbfe3d4, ebp = 0xbfbfe3e0 --- Tracing command csh pid 932 tid 100180 td 0xca3c7000 sched_switch(ca3c7000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3978980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39789b8 sleepq_switch(ca3c7000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39789ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3978a28 sleepq_wait_sig(cbd4fc70,0,f3978a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3978a4c _cv_wait_sig(cbd4fc70,cbd4fc04,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3978a94 tty_wait(cbd4fc00,cbd4fc70,f3978bf0,1,0,...) at tty_wait+0x4b/frame 0xf3978ab8 ttydisc_read(cbd4fc00,f3978bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3978afc ttydev_read(c9615800,f3978bf0,0,0,ca3c7000,...) at ttydev_read+0x55/frame 0xf3978b1c devfs_read_f(c97470a8,f3978bf0,c95f3b00,0,ca3c7000,...) at devfs_read_f+0xb7/frame 0xf3978b54 dofileread(c97470a8,f3978bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3978b88 kern_readv(ca3c7000,10,f3978bf0,c93786b0,bfbfe3c2,...) at kern_readv+0x93/frame 0xf3978bd0 sys_read(ca3c7000,f3978ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xf3978c20 syscall(f3978ce8) at syscall+0x388/frame 0xf3978cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3978cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x2827b553, esp = 0xbfbfe374, ebp = 0xbfbfe388 --- Tracing command su pid 931 tid 100195 td 0xc96ea9c0 sched_switch(c96ea9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf390b968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf390b9a0 sleepq_switch(c96ea9c0,0,c1583a69,1b8,c19cda90,...) at sleepq_switch+0x127/frame 0xf390b9d4 sleepq_catch_signals(c1583a69,16f,14,f0,c96ea9c0,...) at sleepq_catch_signals+0x3b7/frame 0xf390ba10 sleepq_wait_sig(ca2d5000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf390ba34 _sleep(ca2d5000,ca2d50a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf390ba94 kern_wait6(c96ea9c0,0,3a4,0,f390bb2c,...) at kern_wait6+0x47e/frame 0xf390bb00 sys_wait4(c96ea9c0,f390bca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf390bc20 syscall(f390bce8) at syscall+0x388/frame 0xf390bcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf390bcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28185303, esp = 0xbfbfe680, ebp = 0xbfbfe698 --- Tracing command bash pid 924 tid 100151 td 0xc96f59c0 sched_switch(c96f59c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38fc968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf38fc9a0 sleepq_switch(c96f59c0,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf38fc9d4 sleepq_catch_signals(c1583a69,16f,14,a2,c96f59c0,...) at sleepq_catch_signals+0x3b7/frame 0xf38fca10 sleepq_wait_sig(c97286b0,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf38fca34 _sleep(c97286b0,c9728758,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf38fca94 kern_wait6(c96f59c0,7,0,0,f38fcb2c,...) at kern_wait6+0x47e/frame 0xf38fcb00 sys_wait4(c96f59c0,f38fcca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf38fcc20 syscall(f38fcce8) at syscall+0x388/frame 0xf38fccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf38fccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28262303, esp = 0xbfbfe824, ebp = 0xbfbfe83c --- Tracing command sshd pid 923 tid 100185 td 0xca3c6680 sched_switch(ca3c6680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39918e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3991920 sleepq_switch(ca3c6680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf3991954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3991990 sleepq_wait_sig(c96ad164,0,f39919e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xf39919b4 _cv_wait_sig(c96ad164,c96ad150,c1587083,739,c1589e42,...) at _cv_wait_sig+0x231/frame 0xf39919fc seltdwait(ffffffff,0,0,ca3c6680,0,...) at seltdwait+0xbf/frame 0xf3991a30 kern_select(ca3c6680,c,28a17088,28a170c0,0,...) at kern_select+0x904/frame 0xf3991bd8 sys_select(ca3c6680,f3991ca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xf3991c20 syscall(f3991ce8) at syscall+0x388/frame 0xf3991cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3991cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28513543, esp = 0xbfbfe794, ebp = 0xbfbfe7b8 --- Tracing command sshd pid 921 tid 100187 td 0xcaed0340 sched_switch(caed0340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3997920 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3997958 sleepq_switch(caed0340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf399798c sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf39979c8 sleepq_wait_sig(c95fb6e4,0,f3997a20,101,0,...) at sleepq_wait_sig+0x14/frame 0xf39979ec _cv_wait_sig(c95fb6e4,c95fb6d0,c1587083,739,c1587083,...) at _cv_wait_sig+0x231/frame 0xf3997a34 seltdwait(ffffffff,0,0,caed0340,0,...) at seltdwait+0xbf/frame 0xf3997a68 kern_poll(caed0340,bfbfe878,1,0,0,...) at kern_poll+0x4f0/frame 0xf3997be8 sys_poll(caed0340,f3997ca8,c1574eec,4,c119f5d9,...) at sys_poll+0x71/frame 0xf3997c20 syscall(f3997ce8) at syscall+0x388/frame 0xf3997cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3997cdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x284b623b, esp = 0xbfbfe804, ebp = 0xbfbfe820 --- Tracing command login pid 809 tid 100176 td 0xca3c7680 sched_switch(ca3c7680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf396c968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf396c9a0 sleepq_switch(ca3c7680,0,c1583a69,1b8,c19ce190,...) at sleepq_switch+0x127/frame 0xf396c9d4 sleepq_catch_signals(c1583a69,16f,14,150,ca3c7680,...) at sleepq_catch_signals+0x3b7/frame 0xf396ca10 sleepq_wait_sig(c96e7000,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xf396ca34 _sleep(c96e7000,c96e70a8,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xf396ca94 kern_wait6(ca3c7680,0,3b8,0,f396cb2c,...) at kern_wait6+0x47e/frame 0xf396cb00 sys_wait4(ca3c7680,f396cca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xf396cc20 syscall(f396cce8) at syscall+0x388/frame 0xf396ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf396ccdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28187303, esp = 0xbfbfed38, ebp = 0xbfbfed50 --- Tracing command getty pid 808 tid 100175 td 0xc96ea000 sched_switch(c96ea000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3969980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39699b8 sleepq_switch(c96ea000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39699ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3969a28 sleepq_wait_sig(c90f7270,0,f3969a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3969a4c _cv_wait_sig(c90f7270,c90f7204,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3969a94 tty_wait(c90f7200,c90f7270,f3969bf0,1,0,...) at tty_wait+0x4b/frame 0xf3969ab8 ttydisc_read(c90f7200,f3969bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3969afc ttydev_read(c935a800,f3969bf0,0,0,c96ea000,...) at ttydev_read+0x55/frame 0xf3969b1c devfs_read_f(c97812a0,f3969bf0,c97c3e00,0,c96ea000,...) at devfs_read_f+0xb7/frame 0xf3969b54 dofileread(c97812a0,f3969bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3969b88 kern_readv(c96ea000,0,f3969bf0,c96e7358,bfbfede7,...) at kern_readv+0x93/frame 0xf3969bd0 sys_read(c96ea000,f3969ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xf3969c20 syscall(f3969ce8) at syscall+0x388/frame 0xf3969cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3969cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 807 tid 100174 td 0xc96ea340 sched_switch(c96ea340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3966980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39669b8 sleepq_switch(c96ea340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39669ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3966a28 sleepq_wait_sig(c90f7470,0,f3966a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3966a4c _cv_wait_sig(c90f7470,c90f7404,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3966a94 tty_wait(c90f7400,c90f7470,f3966bf0,1,0,...) at tty_wait+0x4b/frame 0xf3966ab8 ttydisc_read(c90f7400,f3966bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3966afc ttydev_read(c935a900,f3966bf0,0,0,c96ea340,...) at ttydev_read+0x55/frame 0xf3966b1c devfs_read_f(c97473b8,f3966bf0,c95f3700,0,c96ea340,...) at devfs_read_f+0xb7/frame 0xf3966b54 dofileread(c97473b8,f3966bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3966b88 kern_readv(c96ea340,0,f3966bf0,c96e76b0,bfbfede7,...) at kern_readv+0x93/frame 0xf3966bd0 sys_read(c96ea340,f3966ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xf3966c20 syscall(f3966ce8) at syscall+0x388/frame 0xf3966cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3966cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 806 tid 100166 td 0xca189340 sched_switch(ca189340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3946980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39469b8 sleepq_switch(ca189340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39469ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3946a28 sleepq_wait_sig(c90f7670,0,f3946a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3946a4c _cv_wait_sig(c90f7670,c90f7604,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3946a94 tty_wait(c90f7600,c90f7670,f3946bf0,1,0,...) at tty_wait+0x4b/frame 0xf3946ab8 ttydisc_read(c90f7600,f3946bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3946afc ttydev_read(c935aa00,f3946bf0,0,0,ca189340,...) at ttydev_read+0x55/frame 0xf3946b1c devfs_read_f(c9746e70,f3946bf0,c95f3a00,0,ca189340,...) at devfs_read_f+0xb7/frame 0xf3946b54 dofileread(c9746e70,f3946bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3946b88 kern_readv(ca189340,0,f3946bf0,c9a98358,bfbfede7,...) at kern_readv+0x93/frame 0xf3946bd0 sys_read(ca189340,f3946ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xf3946c20 syscall(f3946ce8) at syscall+0x388/frame 0xf3946cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3946cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 805 tid 100170 td 0xca1889c0 sched_switch(ca1889c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3959980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39599b8 sleepq_switch(ca1889c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39599ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3959a28 sleepq_wait_sig(c90f7870,0,f3959a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3959a4c _cv_wait_sig(c90f7870,c90f7804,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3959a94 tty_wait(c90f7800,c90f7870,f3959bf0,1,0,...) at tty_wait+0x4b/frame 0xf3959ab8 ttydisc_read(c90f7800,f3959bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3959afc ttydev_read(c935ab00,f3959bf0,0,0,ca1889c0,...) at ttydev_read+0x55/frame 0xf3959b1c devfs_read_f(c9781428,f3959bf0,c8e19500,0,ca1889c0,...) at devfs_read_f+0xb7/frame 0xf3959b54 dofileread(c9781428,f3959bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3959b88 kern_readv(ca1889c0,0,f3959bf0,c9a97a08,bfbfede7,...) at kern_readv+0x93/frame 0xf3959bd0 sys_read(ca1889c0,f3959ca8,c1574eec,4,0,...) at sys_read+0x59/frame 0xf3959c20 syscall(f3959ce8) at syscall+0x388/frame 0xf3959cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3959cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 804 tid 100171 td 0xca188680 sched_switch(ca188680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf395c980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf395c9b8 sleepq_switch(ca188680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf395c9ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf395ca28 sleepq_wait_sig(c90f7a70,0,f395ca80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf395ca4c _cv_wait_sig(c90f7a70,c90f7a04,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf395ca94 tty_wait(c90f7a00,c90f7a70,f395cbf0,1,0,...) at tty_wait+0x4b/frame 0xf395cab8 ttydisc_read(c90f7a00,f395cbf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf395cafc ttydev_read(c935ac00,f395cbf0,0,0,ca188680,...) at ttydev_read+0x55/frame 0xf395cb1c devfs_read_f(c97477a8,f395cbf0,c8e1a600,0,ca188680,...) at devfs_read_f+0xb7/frame 0xf395cb54 dofileread(c97477a8,f395cbf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf395cb88 kern_readv(ca188680,0,f395cbf0,c9a976b0,bfbfede7,...) at kern_readv+0x93/frame 0xf395cbd0 sys_read(ca188680,f395cca8,c1574eec,4,0,...) at sys_read+0x59/frame 0xf395cc20 syscall(f395cce8) at syscall+0x388/frame 0xf395ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf395ccdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 803 tid 100172 td 0xca188340 sched_switch(ca188340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf395f980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf395f9b8 sleepq_switch(ca188340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf395f9ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf395fa28 sleepq_wait_sig(c90f7c70,0,f395fa80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf395fa4c _cv_wait_sig(c90f7c70,c90f7c04,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf395fa94 tty_wait(c90f7c00,c90f7c70,f395fbf0,1,0,...) at tty_wait+0x4b/frame 0xf395fab8 ttydisc_read(c90f7c00,f395fbf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf395fafc ttydev_read(c935ad00,f395fbf0,0,0,ca188340,...) at ttydev_read+0x55/frame 0xf395fb1c devfs_read_f(c9781380,f395fbf0,c97c2800,0,ca188340,...) at devfs_read_f+0xb7/frame 0xf395fb54 dofileread(c9781380,f395fbf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf395fb88 kern_readv(ca188340,0,f395fbf0,c9a97358,bfbfede7,...) at kern_readv+0x93/frame 0xf395fbd0 sys_read(ca188340,f395fca8,c1574eec,4,c19cd380,...) at sys_read+0x59/frame 0xf395fc20 syscall(f395fce8) at syscall+0x388/frame 0xf395fcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf395fcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 802 tid 100173 td 0xca188000 sched_switch(ca188000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3962980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf39629b8 sleepq_switch(ca188000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf39629ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3962a28 sleepq_wait_sig(c90f7e70,0,f3962a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xf3962a4c _cv_wait_sig(c90f7e70,c90f7e04,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xf3962a94 tty_wait(c90f7e00,c90f7e70,f3962bf0,1,0,...) at tty_wait+0x4b/frame 0xf3962ab8 ttydisc_read(c90f7e00,f3962bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xf3962afc ttydev_read(c935ae00,f3962bf0,0,0,ca188000,...) at ttydev_read+0x55/frame 0xf3962b1c devfs_read_f(c9781498,f3962bf0,c97c4000,0,ca188000,...) at devfs_read_f+0xb7/frame 0xf3962b54 dofileread(c9781498,f3962bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xf3962b88 kern_readv(ca188000,0,f3962bf0,c9a97000,bfbfede7,...) at kern_readv+0x93/frame 0xf3962bd0 sys_read(ca188000,f3962ca8,c1574eec,4,0,...) at sys_read+0x59/frame 0xf3962c20 syscall(f3962ce8) at syscall+0x388/frame 0xf3962cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3962cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command getty pid 801 tid 100089 td 0xc96f5000 sched_switch(c96f5000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf23980 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf239b8 sleepq_switch(c96f5000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedf239ec sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedf23a28 sleepq_wait_sig(c90f8070,0,edf23a80,101,0,...) at sleepq_wait_sig+0x14/frame 0xedf23a4c _cv_wait_sig(c90f8070,c90f8004,c158938d,580,1,...) at _cv_wait_sig+0x231/frame 0xedf23a94 tty_wait(c90f8000,c90f8070,edf23bf0,1,0,...) at tty_wait+0x4b/frame 0xedf23ab8 ttydisc_read(c90f8000,edf23bf0,0,ab,c1826574,...) at ttydisc_read+0x366/frame 0xedf23afc ttydev_read(c935b000,edf23bf0,0,0,c96f5000,...) at ttydev_read+0x55/frame 0xedf23b1c devfs_read_f(c9747738,edf23bf0,c96a2600,0,c96f5000,...) at devfs_read_f+0xb7/frame 0xedf23b54 dofileread(c9747738,edf23bf0,ffffffff,ffffffff,0,...) at dofileread+0x97/frame 0xedf23b88 kern_readv(c96f5000,0,edf23bf0,c96e96b0,bfbfede7,...) at kern_readv+0x93/frame 0xedf23bd0 sys_read(c96f5000,edf23ca8,c1574eec,4,c119f5d9,...) at sys_read+0x59/frame 0xedf23c20 syscall(edf23ce8) at syscall+0x388/frame 0xedf23cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf23cdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281d1553, esp = 0xbfbfedb4, ebp = 0xbfbfedc8 --- Tracing command inetd pid 782 tid 100118 td 0xc95bf680 sched_switch(c95bf680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedffd8e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedffd920 sleepq_switch(c95bf680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedffd954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedffd990 sleepq_wait_sig(c96adb24,0,edffd9e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedffd9b4 _cv_wait_sig(c96adb24,c96adb10,c1587083,739,c158713d,...) at _cv_wait_sig+0x231/frame 0xedffd9fc seltdwait(ffffffff,0,0,c95bf680,0,...) at seltdwait+0xbf/frame 0xedffda30 kern_select(c95bf680,6,bfbfe078,0,0,...) at kern_select+0x904/frame 0xedffdbd8 sys_select(c95bf680,edffdca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedffdc20 syscall(edffdce8) at syscall+0x388/frame 0xedffdcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedffdcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281e5543, esp = 0xbfbfe00c, ebp = 0xbfbfe030 --- Tracing command moused pid 752 tid 100157 td 0xc96f5340 sched_switch(c96f5340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39248e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3924920 sleepq_switch(c96f5340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf3924954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3924990 sleepq_wait_sig(c9111124,0,f39249e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xf39249b4 _cv_wait_sig(c9111124,c9111110,c1587083,739,1,...) at _cv_wait_sig+0x231/frame 0xf39249fc seltdwait(ffffffff,0,0,c96f5340,0,...) at seltdwait+0xbf/frame 0xf3924a30 kern_select(c96f5340,400,bfbfebb8,0,0,...) at kern_select+0x904/frame 0xf3924bd8 sys_select(c96f5340,f3924ca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xf3924c20 syscall(f3924ce8) at syscall+0x388/frame 0xf3924cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3924cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281fc543, esp = 0xbfbfea54, ebp = 0xbfbfea78 --- Tracing command cron pid 669 tid 100103 td 0xc953b000 sched_switch(c953b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf63a50 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf63a88 sleepq_switch(c953b000,0,c1583a69,1b8,c1583a69,...) at sleepq_switch+0x127/frame 0xedf63abc sleepq_catch_signals(c0c7e890,c953b000,3,201,3,...) at sleepq_catch_signals+0x3b7/frame 0xedf63af8 sleepq_timedwait_sig(c19bf9c3,6c,16f,bf3752d0,3,...) at sleepq_timedwait_sig+0x14/frame 0xedf63b20 _sleep(c19bf9c3,0,16c,c157ec66,af80e3ca,...) at _sleep+0x347/frame 0xedf63b80 kern_nanosleep(bfbfee28,edf63bf8,edf63c00,0,3b,...) at kern_nanosleep+0x14b/frame 0xedf63be0 sys_nanosleep(c953b000,edf63ca8,c1574eec,4,c97c7768,...) at sys_nanosleep+0x68/frame 0xedf63c20 syscall(edf63ce8) at syscall+0x388/frame 0xedf63cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf63cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281703c3, esp = 0xbfbfeca0, ebp = 0xbfbfecb0 --- Tracing command sendmail pid 664 tid 100106 td 0xc90e39c0 sched_switch(c90e39c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfafa84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedfafabc sleepq_switch(c90e39c0,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xedfafaf0 sleepq_catch_signals(c1583a69,16f,14,2cd,c90e39c0,...) at sleepq_catch_signals+0x3b7/frame 0xedfafb2c sleepq_wait_sig(c8e453bc,74,c120768f,100,0,...) at sleepq_wait_sig+0x14/frame 0xedfafb50 _sleep(c8e453bc,c8e45400,174,c120768f,0,...) at _sleep+0x37e/frame 0xedfafbb0 kern_sigsuspend(c90e39c0,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0xedfafbf0 sys_sigsuspend(c90e39c0,edfafca8,c1574eec,4,c8e45410,...) at sys_sigsuspend+0x55/frame 0xedfafc20 syscall(edfafce8) at syscall+0x388/frame 0xedfafcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfafcdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x28413343, esp = 0xbfbfda9c, ebp = 0xbfbfdaa8 --- Tracing command sendmail pid 661 tid 100155 td 0xca18a340 sched_switch(ca18a340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf391d8dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf391d914 sleepq_switch(ca18a340,0,c1583a69,1b8,c96ad710,...) at sleepq_switch+0x127/frame 0xf391d948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xf391d984 sleepq_timedwait_sig(c96ad724,0,f391d9e8,50000000,0,...) at sleepq_timedwait_sig+0x14/frame 0xf391d9ac _cv_timedwait_sig_sbt(c96ad724,c96ad710,7e9b976f,145,50000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xf391d9fc seltdwait(145,50000000,0,ca18a340,0,...) at seltdwait+0x8d/frame 0xf391da30 kern_select(ca18a340,5,bfbfd140,0,0,...) at kern_select+0x904/frame 0xf391dbd8 sys_select(ca18a340,f391dca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xf391dc20 syscall(f391dce8) at syscall+0x388/frame 0xf391dcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf391dcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28482543, esp = 0xbfbfcff4, ebp = 0xbfbfd018 --- Tracing command sshd pid 658 tid 100156 td 0xca18a000 sched_switch(ca18a000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39208e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3920920 sleepq_switch(ca18a000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf3920954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3920990 sleepq_wait_sig(c9111164,0,f39209e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xf39209b4 _cv_wait_sig(c9111164,c9111150,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xf39209fc seltdwait(ffffffff,0,0,ca18a000,0,...) at seltdwait+0xbf/frame 0xf3920a30 kern_select(ca18a000,7,28a17148,0,0,...) at kern_select+0x904/frame 0xf3920bd8 sys_select(ca18a000,f3920ca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xf3920c20 syscall(f3920ce8) at syscall+0x388/frame 0xf3920cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3920cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28513543, esp = 0xbfbfe86c, ebp = 0xbfbfe890 --- Tracing command smartd pid 621 tid 100153 td 0xca18a680 sched_switch(ca18a680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3908a50 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3908a88 sleepq_switch(ca18a680,0,c1583a69,1b8,c1583a69,...) at sleepq_switch+0x127/frame 0xf3908abc sleepq_catch_signals(c0c7e890,ca18a680,0,201,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3908af8 sleepq_timedwait_sig(c19bf9c0,6c,719,80000000,70,...) at sleepq_timedwait_sig+0x14/frame 0xf3908b20 _sleep(c19bf9c0,0,16c,c157ec66,d52375a0,...) at _sleep+0x347/frame 0xf3908b80 kern_nanosleep(bfbf8bd0,f3908bf8,f3908c00,0,708,...) at kern_nanosleep+0x14b/frame 0xf3908be0 sys_nanosleep(ca18a680,f3908ca8,c1574eec,4,f3908c7c,...) at sys_nanosleep+0x68/frame 0xf3908c20 syscall(f3908ce8) at syscall+0x388/frame 0xf3908cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3908cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x282e43c3, esp = 0xbfbf8ba8, ebp = 0xbfbf8bc0 --- Tracing command powerd pid 602 tid 100096 td 0xc973b9c0 sched_switch(c973b9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf488dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedf48914 sleepq_switch(c973b9c0,0,c1583a69,1b8,c96adb90,...) at sleepq_switch+0x127/frame 0xedf48948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedf48984 sleepq_timedwait_sig(c96adba4,0,edf489e8,3ffffff,0,...) at sleepq_timedwait_sig+0x14/frame 0xedf489ac _cv_timedwait_sig_sbt(c96adba4,c96adb90,c6e8811,141,3ffffff,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedf489fc seltdwait(141,3ffffff,0,c19be910,edf48ad8,...) at seltdwait+0x8d/frame 0xedf48a30 kern_select(c973b9c0,0,bfbfeda0,0,bfbfeda0,...) at kern_select+0x904/frame 0xedf48bd8 sys_select(c973b9c0,edf48ca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedf48c20 syscall(edf48ce8) at syscall+0x388/frame 0xedf48cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf48cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281ce543, esp = 0xbfbfed14, ebp = 0xbfbfed38 --- Tracing command ntpd pid 599 tid 100152 td 0xc96f5680 sched_switch(c96f5680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf39028e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3902920 sleepq_switch(c96f5680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf3902954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf3902990 sleepq_wait_sig(c96adaa4,0,f39029e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xf39029b4 _cv_wait_sig(c96adaa4,c96ada90,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xf39029fc seltdwait(ffffffff,0,0,c96f5680,0,...) at seltdwait+0xbf/frame 0xf3902a30 kern_select(c96f5680,1c,bfbfdee8,0,0,...) at kern_select+0x904/frame 0xf3902bd8 sys_select(c96f5680,f3902ca8,c1574eec,4,c0c1019f,...) at sys_select+0x66/frame 0xf3902c20 syscall(f3902ce8) at syscall+0x388/frame 0xf3902cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3902cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2845e543, esp = 0xbfbfddf4, ebp = 0xbfbfde18 --- Tracing command nfsd pid 545 tid 100119 td 0xc9732000 sched_switch(c9732000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38982d0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3898308 sleepq_switch(c9732000,0,c1583a69,1b8,c91e5100,...) at sleepq_switch+0x127/frame 0xf389833c sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xf3898378 sleepq_timedwait_sig(ca18f390,0,f38983dc,0,0,...) at sleepq_timedwait_sig+0x14/frame 0xf38983a0 _cv_timedwait_sig_sbt(ca18f390,c91e5100,fffffa38,4,0,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xf38983f0 svc_run_internal(c0e9ee60,c91e5100,c9728a08,f3898518,0,...) at svc_run_internal+0xc67/frame 0xf38984d4 svc_run(c91e5000,0,c1565efe,1ed,0,...) at svc_run+0x210/frame 0xf3898528 nfsrvd_nfsd(c9732000,f3898698,c,0,0,...) at nfsrvd_nfsd+0x20c/frame 0xf3898668 nfssvc_nfsd(c9732000,f3898ca8,0,c9732000,f3898cdc,...) at nfssvc_nfsd+0x1e5/frame 0xf3898c08 sys_nfssvc(c9732000,f3898ca8,c1574eec,4,0,...) at sys_nfssvc+0xc4/frame 0xf3898c20 syscall(f3898ce8) at syscall+0x388/frame 0xf3898cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf3898cdc --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280b02df, esp = 0xbfbfe7e4, ebp = 0xbfbfea58 --- Tracing command nfsd pid 545 tid 100120 td 0xc95bf340 sched_switch(c95bf340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf389baac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf389bae4 sleepq_switch(c95bf340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf389bb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf389bb54 sleepq_wait_sig(ca17a190,0,f389bbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf389bb78 _cv_wait_sig(ca17a190,c91e5100,c15b6879,414,0,...) at _cv_wait_sig+0x231/frame 0xf389bbc0 svc_run_internal(f389bcd4,c0bf0ede,c91e5100,f389bce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf389bca4 svc_thread_start(c91e5100,f389bce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf389bcac fork_exit(c0e9ee60,c91e5100,f389bce8) at fork_exit+0x7e/frame 0xf389bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf389bcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100121 td 0xc95bf000 sched_switch(c95bf000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf389eaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf389eae4 sleepq_switch(c95bf000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf389eb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf389eb54 sleepq_wait_sig(ca17e290,0,f389ebac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf389eb78 _cv_wait_sig(ca17e290,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf389ebc0 svc_run_internal(f389ecd4,c0bf0ede,c91e5100,f389ece8,c157546c,...) at svc_run_internal+0xc96/frame 0xf389eca4 svc_thread_start(c91e5100,f389ece8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf389ecac fork_exit(c0e9ee60,c91e5100,f389ece8) at fork_exit+0x7e/frame 0xf389ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf389ecd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100122 td 0xc937c9c0 sched_switch(c937c9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38a1aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38a1ae4 sleepq_switch(c937c9c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38a1b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38a1b54 sleepq_wait_sig(ca17a090,0,f38a1bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38a1b78 _cv_wait_sig(ca17a090,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38a1bc0 svc_run_internal(f38a1cd4,c0bf0ede,c91e5100,f38a1ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38a1ca4 svc_thread_start(c91e5100,f38a1ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38a1cac fork_exit(c0e9ee60,c91e5100,f38a1ce8) at fork_exit+0x7e/frame 0xf38a1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38a1cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100123 td 0xca180680 sched_switch(ca180680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38a4aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38a4ae4 sleepq_switch(ca180680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38a4b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38a4b54 sleepq_wait_sig(ca17e190,0,f38a4bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38a4b78 _cv_wait_sig(ca17e190,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38a4bc0 svc_run_internal(f38a4cd4,c0bf0ede,c91e5100,f38a4ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38a4ca4 svc_thread_start(c91e5100,f38a4ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38a4cac fork_exit(c0e9ee60,c91e5100,f38a4ce8) at fork_exit+0x7e/frame 0xf38a4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38a4cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100124 td 0xca180340 sched_switch(ca180340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38a7aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38a7ae4 sleepq_switch(ca180340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38a7b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38a7b54 sleepq_wait_sig(ca178e90,0,f38a7bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38a7b78 _cv_wait_sig(ca178e90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38a7bc0 svc_run_internal(f38a7cd4,c0bf0ede,c91e5100,f38a7ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38a7ca4 svc_thread_start(c91e5100,f38a7ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38a7cac fork_exit(c0e9ee60,c91e5100,f38a7ce8) at fork_exit+0x7e/frame 0xf38a7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38a7cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100125 td 0xca180000 sched_switch(ca180000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38aaaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38aaae4 sleepq_switch(ca180000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38aab18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38aab54 sleepq_wait_sig(ca178d90,0,f38aabac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38aab78 _cv_wait_sig(ca178d90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38aabc0 svc_run_internal(f38aacd4,c0bf0ede,c91e5100,f38aace8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38aaca4 svc_thread_start(c91e5100,f38aace8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38aacac fork_exit(c0e9ee60,c91e5100,f38aace8) at fork_exit+0x7e/frame 0xf38aacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38aacd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100126 td 0xca17f9c0 sched_switch(ca17f9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38adaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38adae4 sleepq_switch(ca17f9c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38adb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38adb54 sleepq_wait_sig(ca178c90,0,f38adbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38adb78 _cv_wait_sig(ca178c90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38adbc0 svc_run_internal(f38adcd4,c0bf0ede,c91e5100,f38adce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38adca4 svc_thread_start(c91e5100,f38adce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38adcac fork_exit(c0e9ee60,c91e5100,f38adce8) at fork_exit+0x7e/frame 0xf38adcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38adcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100127 td 0xca17f680 sched_switch(ca17f680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38b0aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38b0ae4 sleepq_switch(ca17f680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38b0b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38b0b54 sleepq_wait_sig(ca17e090,0,f38b0bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38b0b78 _cv_wait_sig(ca17e090,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38b0bc0 svc_run_internal(f38b0cd4,c0bf0ede,c91e5100,f38b0ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38b0ca4 svc_thread_start(c91e5100,f38b0ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38b0cac fork_exit(c0e9ee60,c91e5100,f38b0ce8) at fork_exit+0x7e/frame 0xf38b0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38b0cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100128 td 0xca17f340 sched_switch(ca17f340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38b3aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38b3ae4 sleepq_switch(ca17f340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38b3b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38b3b54 sleepq_wait_sig(ca178b90,0,f38b3bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38b3b78 _cv_wait_sig(ca178b90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38b3bc0 svc_run_internal(f38b3cd4,c0bf0ede,c91e5100,f38b3ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38b3ca4 svc_thread_start(c91e5100,f38b3ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38b3cac fork_exit(c0e9ee60,c91e5100,f38b3ce8) at fork_exit+0x7e/frame 0xf38b3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38b3cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100129 td 0xca17f000 sched_switch(ca17f000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38b6aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38b6ae4 sleepq_switch(ca17f000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38b6b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38b6b54 sleepq_wait_sig(ca17de90,0,f38b6bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38b6b78 _cv_wait_sig(ca17de90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38b6bc0 svc_run_internal(f38b6cd4,c0bf0ede,c91e5100,f38b6ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38b6ca4 svc_thread_start(c91e5100,f38b6ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38b6cac fork_exit(c0e9ee60,c91e5100,f38b6ce8) at fork_exit+0x7e/frame 0xf38b6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38b6cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100130 td 0xca17b9c0 sched_switch(ca17b9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38b9aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38b9ae4 sleepq_switch(ca17b9c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38b9b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38b9b54 sleepq_wait_sig(ca178a90,0,f38b9bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38b9b78 _cv_wait_sig(ca178a90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38b9bc0 svc_run_internal(f38b9cd4,c0bf0ede,c91e5100,f38b9ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38b9ca4 svc_thread_start(c91e5100,f38b9ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38b9cac fork_exit(c0e9ee60,c91e5100,f38b9ce8) at fork_exit+0x7e/frame 0xf38b9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38b9cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100131 td 0xca17b680 sched_switch(ca17b680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38bcaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38bcae4 sleepq_switch(ca17b680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38bcb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38bcb54 sleepq_wait_sig(ca17dd90,0,f38bcbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38bcb78 _cv_wait_sig(ca17dd90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38bcbc0 svc_run_internal(f38bccd4,c0bf0ede,c91e5100,f38bcce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38bcca4 svc_thread_start(c91e5100,f38bcce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38bccac fork_exit(c0e9ee60,c91e5100,f38bcce8) at fork_exit+0x7e/frame 0xf38bccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38bccd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100132 td 0xca17b340 sched_switch(ca17b340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38bfaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38bfae4 sleepq_switch(ca17b340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38bfb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38bfb54 sleepq_wait_sig(ca178990,0,f38bfbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38bfb78 _cv_wait_sig(ca178990,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38bfbc0 svc_run_internal(f38bfcd4,c0bf0ede,c91e5100,f38bfce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38bfca4 svc_thread_start(c91e5100,f38bfce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38bfcac fork_exit(c0e9ee60,c91e5100,f38bfce8) at fork_exit+0x7e/frame 0xf38bfcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38bfcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100133 td 0xca17b000 sched_switch(ca17b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38c2aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38c2ae4 sleepq_switch(ca17b000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38c2b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38c2b54 sleepq_wait_sig(ca178890,0,f38c2bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38c2b78 _cv_wait_sig(ca178890,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38c2bc0 svc_run_internal(f38c2cd4,c0bf0ede,c91e5100,f38c2ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38c2ca4 svc_thread_start(c91e5100,f38c2ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38c2cac fork_exit(c0e9ee60,c91e5100,f38c2ce8) at fork_exit+0x7e/frame 0xf38c2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38c2cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100134 td 0xc973c9c0 sched_switch(c973c9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38c5aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38c5ae4 sleepq_switch(c973c9c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38c5b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38c5b54 sleepq_wait_sig(ca17dc90,0,f38c5bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38c5b78 _cv_wait_sig(ca17dc90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38c5bc0 svc_run_internal(f38c5cd4,c0bf0ede,c91e5100,f38c5ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38c5ca4 svc_thread_start(c91e5100,f38c5ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38c5cac fork_exit(c0e9ee60,c91e5100,f38c5ce8) at fork_exit+0x7e/frame 0xf38c5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38c5cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100135 td 0xc973c680 sched_switch(c973c680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38c8aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38c8ae4 sleepq_switch(c973c680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38c8b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38c8b54 sleepq_wait_sig(ca178790,0,f38c8bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38c8b78 _cv_wait_sig(ca178790,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38c8bc0 svc_run_internal(f38c8cd4,c0bf0ede,c91e5100,f38c8ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38c8ca4 svc_thread_start(c91e5100,f38c8ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38c8cac fork_exit(c0e9ee60,c91e5100,f38c8ce8) at fork_exit+0x7e/frame 0xf38c8cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38c8cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100136 td 0xca1879c0 sched_switch(ca1879c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38cbaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38cbae4 sleepq_switch(ca1879c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38cbb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38cbb54 sleepq_wait_sig(ca178690,0,f38cbbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38cbb78 _cv_wait_sig(ca178690,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38cbbc0 svc_run_internal(f38cbcd4,c0bf0ede,c91e5100,f38cbce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38cbca4 svc_thread_start(c91e5100,f38cbce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38cbcac fork_exit(c0e9ee60,c91e5100,f38cbce8) at fork_exit+0x7e/frame 0xf38cbcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38cbcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100137 td 0xca187680 sched_switch(ca187680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38ceaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38ceae4 sleepq_switch(ca187680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38ceb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38ceb54 sleepq_wait_sig(ca17db90,0,f38cebac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38ceb78 _cv_wait_sig(ca17db90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38cebc0 svc_run_internal(f38cecd4,c0bf0ede,c91e5100,f38cece8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38ceca4 svc_thread_start(c91e5100,f38cece8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38cecac fork_exit(c0e9ee60,c91e5100,f38cece8) at fork_exit+0x7e/frame 0xf38cecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38cecd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100138 td 0xca187340 sched_switch(ca187340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38d1aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38d1ae4 sleepq_switch(ca187340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38d1b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38d1b54 sleepq_wait_sig(ca17da90,0,f38d1bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38d1b78 _cv_wait_sig(ca17da90,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38d1bc0 svc_run_internal(f38d1cd4,c0bf0ede,c91e5100,f38d1ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38d1ca4 svc_thread_start(c91e5100,f38d1ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38d1cac fork_exit(c0e9ee60,c91e5100,f38d1ce8) at fork_exit+0x7e/frame 0xf38d1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38d1cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100139 td 0xca187000 sched_switch(ca187000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38d4aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38d4ae4 sleepq_switch(ca187000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38d4b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38d4b54 sleepq_wait_sig(ca178590,0,f38d4bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38d4b78 _cv_wait_sig(ca178590,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38d4bc0 svc_run_internal(f38d4cd4,c0bf0ede,c91e5100,f38d4ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38d4ca4 svc_thread_start(c91e5100,f38d4ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38d4cac fork_exit(c0e9ee60,c91e5100,f38d4ce8) at fork_exit+0x7e/frame 0xf38d4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38d4cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100140 td 0xca1869c0 sched_switch(ca1869c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38d7aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38d7ae4 sleepq_switch(ca1869c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38d7b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38d7b54 sleepq_wait_sig(ca17d990,0,f38d7bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38d7b78 _cv_wait_sig(ca17d990,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38d7bc0 svc_run_internal(f38d7cd4,c0bf0ede,c91e5100,f38d7ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38d7ca4 svc_thread_start(c91e5100,f38d7ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38d7cac fork_exit(c0e9ee60,c91e5100,f38d7ce8) at fork_exit+0x7e/frame 0xf38d7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38d7cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100141 td 0xca186680 sched_switch(ca186680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38daaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38daae4 sleepq_switch(ca186680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38dab18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38dab54 sleepq_wait_sig(ca178490,0,f38dabac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38dab78 _cv_wait_sig(ca178490,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38dabc0 svc_run_internal(f38dacd4,c0bf0ede,c91e5100,f38dace8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38daca4 svc_thread_start(c91e5100,f38dace8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38dacac fork_exit(c0e9ee60,c91e5100,f38dace8) at fork_exit+0x7e/frame 0xf38dacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38dacd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100142 td 0xca186340 sched_switch(ca186340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38ddaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38ddae4 sleepq_switch(ca186340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38ddb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38ddb54 sleepq_wait_sig(ca178390,0,f38ddbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38ddb78 _cv_wait_sig(ca178390,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38ddbc0 svc_run_internal(f38ddcd4,c0bf0ede,c91e5100,f38ddce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38ddca4 svc_thread_start(c91e5100,f38ddce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38ddcac fork_exit(c0e9ee60,c91e5100,f38ddce8) at fork_exit+0x7e/frame 0xf38ddcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38ddcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100143 td 0xca186000 sched_switch(ca186000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38e0aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38e0ae4 sleepq_switch(ca186000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38e0b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38e0b54 sleepq_wait_sig(ca17d890,0,f38e0bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38e0b78 _cv_wait_sig(ca17d890,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38e0bc0 svc_run_internal(f38e0cd4,c0bf0ede,c91e5100,f38e0ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38e0ca4 svc_thread_start(c91e5100,f38e0ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38e0cac fork_exit(c0e9ee60,c91e5100,f38e0ce8) at fork_exit+0x7e/frame 0xf38e0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38e0cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100144 td 0xca1859c0 sched_switch(ca1859c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38e3aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38e3ae4 sleepq_switch(ca1859c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38e3b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38e3b54 sleepq_wait_sig(ca17d790,0,f38e3bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38e3b78 _cv_wait_sig(ca17d790,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38e3bc0 svc_run_internal(f38e3cd4,c0bf0ede,c91e5100,f38e3ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38e3ca4 svc_thread_start(c91e5100,f38e3ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38e3cac fork_exit(c0e9ee60,c91e5100,f38e3ce8) at fork_exit+0x7e/frame 0xf38e3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38e3cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100145 td 0xca185680 sched_switch(ca185680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38e6aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38e6ae4 sleepq_switch(ca185680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38e6b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38e6b54 sleepq_wait_sig(ca17d690,0,f38e6bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38e6b78 _cv_wait_sig(ca17d690,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38e6bc0 svc_run_internal(f38e6cd4,c0bf0ede,c91e5100,f38e6ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38e6ca4 svc_thread_start(c91e5100,f38e6ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38e6cac fork_exit(c0e9ee60,c91e5100,f38e6ce8) at fork_exit+0x7e/frame 0xf38e6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38e6cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100146 td 0xca185340 sched_switch(ca185340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38e9aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38e9ae4 sleepq_switch(ca185340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38e9b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38e9b54 sleepq_wait_sig(ca17d590,0,f38e9bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38e9b78 _cv_wait_sig(ca17d590,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38e9bc0 svc_run_internal(f38e9cd4,c0bf0ede,c91e5100,f38e9ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38e9ca4 svc_thread_start(c91e5100,f38e9ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38e9cac fork_exit(c0e9ee60,c91e5100,f38e9ce8) at fork_exit+0x7e/frame 0xf38e9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38e9cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100147 td 0xca185000 sched_switch(ca185000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38ecaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38ecae4 sleepq_switch(ca185000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38ecb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38ecb54 sleepq_wait_sig(ca17d490,0,f38ecbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38ecb78 _cv_wait_sig(ca17d490,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38ecbc0 svc_run_internal(f38eccd4,c0bf0ede,c91e5100,f38ecce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38ecca4 svc_thread_start(c91e5100,f38ecce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38eccac fork_exit(c0e9ee60,c91e5100,f38ecce8) at fork_exit+0x7e/frame 0xf38eccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38eccd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100148 td 0xca1809c0 sched_switch(ca1809c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38efaac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38efae4 sleepq_switch(ca1809c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38efb18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38efb54 sleepq_wait_sig(ca178290,0,f38efbac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38efb78 _cv_wait_sig(ca178290,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38efbc0 svc_run_internal(f38efcd4,c0bf0ede,c91e5100,f38efce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38efca4 svc_thread_start(c91e5100,f38efce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38efcac fork_exit(c0e9ee60,c91e5100,f38efce8) at fork_exit+0x7e/frame 0xf38efcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38efcd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100149 td 0xca18b000 sched_switch(ca18b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38f2aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38f2ae4 sleepq_switch(ca18b000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38f2b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38f2b54 sleepq_wait_sig(ca17d390,0,f38f2bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38f2b78 _cv_wait_sig(ca17d390,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38f2bc0 svc_run_internal(f38f2cd4,c0bf0ede,c91e5100,f38f2ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38f2ca4 svc_thread_start(c91e5100,f38f2ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38f2cac fork_exit(c0e9ee60,c91e5100,f38f2ce8) at fork_exit+0x7e/frame 0xf38f2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38f2cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 545 tid 100150 td 0xca18a9c0 sched_switch(ca18a9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf38f5aac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xf38f5ae4 sleepq_switch(ca18a9c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xf38f5b18 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xf38f5b54 sleepq_wait_sig(ca17d290,0,f38f5bac,101,0,...) at sleepq_wait_sig+0x14/frame 0xf38f5b78 _cv_wait_sig(ca17d290,c91e5100,c15b6879,45c,0,...) at _cv_wait_sig+0x231/frame 0xf38f5bc0 svc_run_internal(f38f5cd4,c0bf0ede,c91e5100,f38f5ce8,c157546c,...) at svc_run_internal+0xc96/frame 0xf38f5ca4 svc_thread_start(c91e5100,f38f5ce8,c157546c,402,0,...) at svc_thread_start+0xd/frame 0xf38f5cac fork_exit(c0e9ee60,c91e5100,f38f5ce8) at fork_exit+0x7e/frame 0xf38f5cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf38f5cd4 --- trap 0x20, eip = 0xc, esp = 0x33, ebp = 0xbfbfe937 --- Tracing command nfsd pid 538 tid 100113 td 0xc9732680 sched_switch(c9732680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfe88e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedfe8920 sleepq_switch(c9732680,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedfe8954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedfe8990 sleepq_wait_sig(c96ae024,0,edfe89e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedfe89b4 _cv_wait_sig(c96ae024,c96ae010,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xedfe89fc seltdwait(ffffffff,0,0,c9732680,0,...) at seltdwait+0xbf/frame 0xedfe8a30 kern_select(c9732680,7,bfbfec30,0,0,...) at kern_select+0x904/frame 0xedfe8bd8 sys_select(c9732680,edfe8ca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedfe8c20 syscall(edfe8ce8) at syscall+0x388/frame 0xedfe8cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfe8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bc543, esp = 0xbfbfea34, ebp = 0xbfbfea58 --- Tracing command mountd pid 532 tid 100109 td 0xc96f3340 sched_switch(c96f3340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfd98e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedfd9920 sleepq_switch(c96f3340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedfd9954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedfd9990 sleepq_wait_sig(c9538c64,0,edfd99e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedfd99b4 _cv_wait_sig(c9538c64,c9538c50,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xedfd99fc seltdwait(ffffffff,0,0,c96f3340,0,...) at seltdwait+0xbf/frame 0xedfd9a30 kern_select(c96f3340,9,bfbfe848,0,0,...) at kern_select+0x904/frame 0xedfd9bd8 sys_select(c96f3340,edfd9ca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedfd9c20 syscall(edfd9ce8) at syscall+0x388/frame 0xedfd9cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfd9cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281d5543, esp = 0xbfbfe7d4, ebp = 0xbfbfe7f8 --- Tracing command nfsuserd pid 525 tid 100117 td 0xc95bf9c0 sched_switch(c95bf9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedff88dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedff8914 sleepq_switch(c95bf9c0,0,c1583a69,1b8,c96ae650,...) at sleepq_switch+0x127/frame 0xedff8948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedff8984 sleepq_timedwait_sig(c96ae664,0,edff89e8,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedff89ac _cv_timedwait_sig_sbt(c96ae664,c96ae650,dc328874,157,e0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedff89fc seltdwait(157,e0000000,1,c95bf9c0,0,...) at seltdwait+0x8d/frame 0xedff8a30 kern_select(c95bf9c0,4,bfbfdc08,0,0,...) at kern_select+0x904/frame 0xedff8bd8 sys_select(c95bf9c0,edff8ca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xedff8c20 syscall(edff8ce8) at syscall+0x388/frame 0xedff8cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedff8cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bb543, esp = 0xbfbfdb54, ebp = 0xbfbfdca0 --- Tracing command nfsuserd pid 524 tid 100116 td 0xc953a000 sched_switch(c953a000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedff48dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedff4914 sleepq_switch(c953a000,0,c1583a69,1b8,c9537790,...) at sleepq_switch+0x127/frame 0xedff4948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedff4984 sleepq_timedwait_sig(c95377a4,0,edff49e8,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedff49ac _cv_timedwait_sig_sbt(c95377a4,c9537790,31546fd9,158,e0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedff49fc seltdwait(158,e0000000,1,c953a000,0,...) at seltdwait+0x8d/frame 0xedff4a30 kern_select(c953a000,4,bfbfdc08,0,0,...) at kern_select+0x904/frame 0xedff4bd8 sys_select(c953a000,edff4ca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xedff4c20 syscall(edff4ce8) at syscall+0x388/frame 0xedff4cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedff4cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bb543, esp = 0xbfbfdb54, ebp = 0xbfbfdca0 --- Tracing command nfsuserd pid 523 tid 100115 td 0xc953a340 sched_switch(c953a340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedff08dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedff0914 sleepq_switch(c953a340,0,c1583a69,1b8,c95fc510,...) at sleepq_switch+0x127/frame 0xedff0948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedff0984 sleepq_timedwait_sig(c95fc524,0,edff09e8,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedff09ac _cv_timedwait_sig_sbt(c95fc524,c95fc510,dc328874,157,e0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedff09fc seltdwait(157,e0000000,1,c953a340,0,...) at seltdwait+0x8d/frame 0xedff0a30 kern_select(c953a340,4,bfbfdc08,0,0,...) at kern_select+0x904/frame 0xedff0bd8 sys_select(c953a340,edff0ca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xedff0c20 syscall(edff0ce8) at syscall+0x388/frame 0xedff0cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedff0cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bb543, esp = 0xbfbfdb54, ebp = 0xbfbfdca0 --- Tracing command nfsuserd pid 522 tid 100110 td 0xc953a680 sched_switch(c953a680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfdd8dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedfdd914 sleepq_switch(c953a680,0,c1583a69,1b8,c9537710,...) at sleepq_switch+0x127/frame 0xedfdd948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedfdd984 sleepq_timedwait_sig(c9537724,0,edfdd9e8,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedfdd9ac _cv_timedwait_sig_sbt(c9537724,c9537710,31546fd9,158,e0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedfdd9fc seltdwait(158,e0000000,1,c953a680,0,...) at seltdwait+0x8d/frame 0xedfdda30 kern_select(c953a680,4,bfbfdc08,0,0,...) at kern_select+0x904/frame 0xedfddbd8 sys_select(c953a680,edfddca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedfddc20 syscall(edfddce8) at syscall+0x388/frame 0xedfddcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfddcdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281bb543, esp = 0xbfbfdb54, ebp = 0xbfbfdca0 --- Tracing command nfsuserd pid 521 tid 100114 td 0xc9732340 sched_switch(c9732340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfeca84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedfecabc sleepq_switch(c9732340,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xedfecaf0 sleepq_catch_signals(c1583a69,16f,14,2dc,c9732340,...) at sleepq_catch_signals+0x3b7/frame 0xedfecb2c sleepq_wait_sig(c9729064,74,c120768f,100,0,...) at sleepq_wait_sig+0x14/frame 0xedfecb50 _sleep(c9729064,c97290a8,174,c120768f,0,...) at _sleep+0x37e/frame 0xedfecbb0 kern_sigsuspend(c9732340,0,0,0,0,...) at kern_sigsuspend+0xd4/frame 0xedfecbf0 sys_sigsuspend(c9732340,edfecca8,c1574eec,4,0,...) at sys_sigsuspend+0x55/frame 0xedfecc20 syscall(edfecce8) at syscall+0x388/frame 0xedfeccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfeccdc --- syscall (4, FreeBSD ELF32, sys_write), eip = 0x2814c343, esp = 0xbfbfdc94, ebp = 0xbfbfdca0 --- Tracing command casperd pid 504 tid 100111 td 0xc97329c0 sched_switch(c97329c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfe28e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedfe2920 sleepq_switch(c97329c0,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedfe2954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedfe2990 sleepq_wait_sig(c9537864,0,edfe29e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedfe29b4 _cv_wait_sig(c9537864,c9537850,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xedfe29fc seltdwait(ffffffff,0,0,c97329c0,0,...) at seltdwait+0xbf/frame 0xedfe2a30 kern_select(c97329c0,7,bfbfe9c8,0,0,...) at kern_select+0x904/frame 0xedfe2bd8 sys_select(c97329c0,edfe2ca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xedfe2c20 syscall(edfe2ce8) at syscall+0x388/frame 0xedfe2cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfe2cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281ec543, esp = 0xbfbfe954, ebp = 0xbfbfe978 --- Tracing command casperd pid 503 tid 100107 td 0xc973b000 sched_switch(c973b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfca8e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedfca920 sleepq_switch(c973b000,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedfca954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedfca990 sleepq_wait_sig(c95378a4,0,edfca9e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedfca9b4 _cv_wait_sig(c95378a4,c9537890,c1587083,739,c0c9f41d,...) at _cv_wait_sig+0x231/frame 0xedfca9fc seltdwait(ffffffff,0,0,c973b000,0,...) at seltdwait+0xbf/frame 0xedfcaa30 kern_select(c973b000,6,bfbfe850,0,0,...) at kern_select+0x904/frame 0xedfcabd8 sys_select(c973b000,edfcaca8,c1574eec,4,0,...) at sys_select+0x66/frame 0xedfcac20 syscall(edfcace8) at syscall+0x388/frame 0xedfcacdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfcacdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281ec543, esp = 0xbfbfe7f4, ebp = 0xbfbfe818 --- Tracing command rpcbind pid 489 tid 100104 td 0xc96f39c0 sched_switch(c96f39c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf6c914 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedf6c94c sleepq_switch(c96f39c0,0,c1583a69,1b8,c9537810,...) at sleepq_switch+0x127/frame 0xedf6c980 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedf6c9bc sleepq_timedwait_sig(c9537824,0,edf6ca20,e0000000,1,...) at sleepq_timedwait_sig+0x14/frame 0xedf6c9e4 _cv_timedwait_sig_sbt(c9537824,c9537810,b8349a2b,158,e0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedf6ca34 seltdwait(158,e0000000,1,c96f39c0,0,...) at seltdwait+0x8d/frame 0xedf6ca68 kern_poll(c96f39c0,bfbfccc0,7,edf6cc08,0,...) at kern_poll+0x4f0/frame 0xedf6cbe8 sys_poll(c96f39c0,edf6cca8,c1574eec,4,c119f5d9,...) at sys_poll+0x71/frame 0xedf6cc20 syscall(edf6cce8) at syscall+0x388/frame 0xedf6ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf6ccdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2816c23b, esp = 0xbfbfcbe4, ebp = 0xbfbfcbf8 --- Tracing command watchdogd pid 472 tid 100108 td 0xc96f3680 sched_switch(c96f3680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedfd1a50 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedfd1a88 sleepq_switch(c96f3680,0,c1583a69,1b8,c1583a69,...) at sleepq_switch+0x127/frame 0xedfd1abc sleepq_catch_signals(c0c7e890,c96f3680,2,201,2,...) at sleepq_catch_signals+0x3b7/frame 0xedfd1af8 sleepq_timedwait_sig(c19bf9c2,6c,179,c0000000,3,...) at sleepq_timedwait_sig+0x14/frame 0xedfd1b20 _sleep(c19bf9c2,0,16c,c157ec66,c3373876,...) at _sleep+0x347/frame 0xedfd1b80 kern_nanosleep(bfbfece0,edfd1bf8,edfd1c00,0,3c,...) at kern_nanosleep+0x14b/frame 0xedfd1be0 sys_nanosleep(c96f3680,edfd1ca8,c1574eec,4,0,...) at sys_nanosleep+0x68/frame 0xedfd1c20 syscall(edfd1ce8) at syscall+0x388/frame 0xedfd1cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedfd1cdc --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x2815f3c3, esp = 0xbfbfecd4, ebp = 0xbfbfed18 --- Tracing command syslogd pid 468 tid 100097 td 0xc953b340 sched_switch(c953b340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf4c8e8 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf4c920 sleepq_switch(c953b340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedf4c954 sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedf4c990 sleepq_wait_sig(c95378e4,0,edf4c9e8,101,0,...) at sleepq_wait_sig+0x14/frame 0xedf4c9b4 _cv_wait_sig(c95378e4,c95378d0,c1587083,739,1,...) at _cv_wait_sig+0x231/frame 0xedf4c9fc seltdwait(ffffffff,0,0,c953b340,0,...) at seltdwait+0xbf/frame 0xedf4ca30 kern_select(c953b340,9,286220d8,0,0,...) at kern_select+0x904/frame 0xedf4cbd8 sys_select(c953b340,edf4cca8,c1574eec,4,c119f5d9,...) at sys_select+0x66/frame 0xedf4cc20 syscall(edf4cce8) at syscall+0x388/frame 0xedf4ccdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf4ccdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281d4543, esp = 0xbfbfdfc4, ebp = 0xbfbfdfe8 --- Tracing command rtsold pid 359 tid 100102 td 0xc973b340 sched_switch(c973b340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf5f920 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf5f958 sleepq_switch(c973b340,0,c1583a69,1b8,b0,...) at sleepq_switch+0x127/frame 0xedf5f98c sleepq_catch_signals(b0,0,14,af,0,...) at sleepq_catch_signals+0x3b7/frame 0xedf5f9c8 sleepq_wait_sig(c95386a4,0,edf5fa20,101,0,...) at sleepq_wait_sig+0x14/frame 0xedf5f9ec _cv_wait_sig(c95386a4,c9538690,c1587083,739,c1587083,...) at _cv_wait_sig+0x231/frame 0xedf5fa34 seltdwait(ffffffff,0,0,c973b340,0,...) at seltdwait+0xbf/frame 0xedf5fa68 kern_poll(c973b340,bfbfee10,2,0,0,...) at kern_poll+0x4f0/frame 0xedf5fbe8 sys_poll(c973b340,edf5fca8,c1574eec,4,0,...) at sys_poll+0x71/frame 0xedf5fc20 syscall(edf5fce8) at syscall+0x388/frame 0xedf5fcdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf5fcdc --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2816223b, esp = 0xbfbfedac, ebp = 0xbfbfedc0 --- Tracing command devd pid 318 tid 100092 td 0xc937a340 sched_switch(c937a340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf318dc mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedf31914 sleepq_switch(c937a340,0,c1583a69,1b8,c95fb7d0,...) at sleepq_switch+0x127/frame 0xedf31948 sleepq_catch_signals(b0,b0,0,14,af,...) at sleepq_catch_signals+0x3b7/frame 0xedf31984 sleepq_timedwait_sig(c95fb7e4,0,edf319e8,c0000000,3,...) at sleepq_timedwait_sig+0x14/frame 0xedf319ac _cv_timedwait_sig_sbt(c95fb7e4,c95fb7d0,f3ee353,158,c0000000,...) at _cv_timedwait_sig_sbt+0x25a/frame 0xedf319fc seltdwait(158,c0000000,3,c937a340,0,...) at seltdwait+0x8d/frame 0xedf31a30 kern_select(c937a340,6,bfbfcdc0,0,0,...) at kern_select+0x904/frame 0xedf31bd8 sys_select(c937a340,edf31ca8,c1574eec,4,c19cda90,...) at sys_select+0x66/frame 0xedf31c20 syscall(edf31ce8) at syscall+0x388/frame 0xedf31cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xedf31cdc --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x80e83a3, esp = 0xbfbfcd44, ebp = 0xbfbfee54 --- Tracing command syncer pid 23 tid 100087 td 0xc937a680 sched_switch(c937a680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf1cb58 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf1cb90 sleepq_switch(c937a680,0,c1583a69,29c,0,...) at sleepq_switch+0x127/frame 0xedf1cbc4 sleepq_timedwait(c1a39d10,0,edf1cc24,0,0,...) at sleepq_timedwait+0x48/frame 0xedf1cbe8 _cv_timedwait_sbt(c1a39d10,c1a39cf0,fffffed8,0,0,...) at _cv_timedwait_sbt+0x25a/frame 0xedf1cc38 sched_sync(0,edf1cce8,c157546c,402,0,...) at sched_sync+0x6b6/frame 0xedf1ccac fork_exit(c0cf5de0,0,edf1cce8) at fork_exit+0x7e/frame 0xedf1ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf1ccd4 --- trap 0, eip = 0, esp = 0xedf1cd20, ebp = 0 --- Tracing command vnlru pid 22 tid 100086 td 0xc937a9c0 sched_switch(c937a9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf19b48 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf19b80 sleepq_switch(c937a9c0,0,c1583a69,29c,c937a9c0,...) at sleepq_switch+0x127/frame 0xedf19bb4 sleepq_timedwait(c9376358,60,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf19bd8 _sleep(c9376358,c1a39cbc,260,c1590654,fffffed8,...) at _sleep+0x362/frame 0xedf19c38 vnlru_proc(0,edf19ce8,c157546c,402,0,...) at vnlru_proc+0xbf/frame 0xedf19cac fork_exit(c0cf54d0,0,edf19ce8) at fork_exit+0x7e/frame 0xedf19cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf19cd4 --- trap 0, eip = 0, esp = 0xedf19d20, ebp = 0 --- Tracing command bufdaemon pid 21 tid 100085 td 0xc937b000 sched_switch(c937b000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf16b88 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf16bc0 sleepq_switch(c937b000,0,c1583a69,29c,c937b000,...) at sleepq_switch+0x127/frame 0xedf16bf4 sleepq_timedwait(c1a38b04,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf16c18 _sleep(c1a38b04,c1a38a80,54,c158d87a,fffffed8,...) at _sleep+0x362/frame 0xedf16c78 buf_daemon(0,edf16ce8,c157546c,402,0,...) at buf_daemon+0x8c/frame 0xedf16cac fork_exit(c0cd78f0,0,edf16ce8) at fork_exit+0x7e/frame 0xedf16cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf16cd4 --- trap 0, eip = 0, esp = 0xedf16d20, ebp = 0 --- Tracing command bufdaemon pid 21 tid 100098 td 0xc96f4680 sched_switch(c96f4680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf50b78 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf50bb0 sleepq_switch(c96f4680,0,c1583a69,29c,c96f4680,...) at sleepq_switch+0x127/frame 0xedf50be4 sleepq_timedwait(c9a1c284,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf50c08 _sleep(c9a1c284,c9a1c200,54,c15bdd59,7fffff6c,...) at _sleep+0x362/frame 0xedf50c68 softdep_flush(c95eb2ec,edf50ce8,c157546c,402,0,...) at softdep_flush+0x192/frame 0xedf50cac fork_exit(c0ee3b30,c95eb2ec,edf50ce8) at fork_exit+0x7e/frame 0xedf50cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf50cd4 --- trap 0, eip = 0, esp = 0xedf50d20, ebp = 0 --- Tracing command bufdaemon pid 21 tid 100100 td 0xc96f4000 sched_switch(c96f4000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf56b78 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf56bb0 sleepq_switch(c96f4000,0,c1583a69,29c,c96f4000,...) at sleepq_switch+0x127/frame 0xedf56be4 sleepq_timedwait(c9a1be84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf56c08 _sleep(c9a1be84,c9a1be00,54,c15bdd59,7fffff6c,...) at _sleep+0x362/frame 0xedf56c68 softdep_flush(c95eabb0,edf56ce8,c157546c,402,0,...) at softdep_flush+0x192/frame 0xedf56cac fork_exit(c0ee3b30,c95eabb0,edf56ce8) at fork_exit+0x7e/frame 0xedf56cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf56cd4 --- trap 0, eip = 0, esp = 0xedf56d20, ebp = 0 --- Tracing command bufdaemon pid 21 tid 100101 td 0xc973b680 sched_switch(c973b680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf59b78 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf59bb0 sleepq_switch(c973b680,0,c1583a69,29c,c973b680,...) at sleepq_switch+0x127/frame 0xedf59be4 sleepq_timedwait(c9afec84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf59c08 _sleep(c9afec84,c9afec00,54,c15bdd59,7fffff6c,...) at _sleep+0x362/frame 0xedf59c68 softdep_flush(c95ea8c4,edf59ce8,c157546c,402,0,...) at softdep_flush+0x192/frame 0xedf59cac fork_exit(c0ee3b30,c95ea8c4,edf59ce8) at fork_exit+0x7e/frame 0xedf59cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf59cd4 --- trap 0, eip = 0, esp = 0xedf59d20, ebp = 0 --- Tracing command bufdaemon pid 21 tid 100167 td 0xca2ab340 sched_switch(ca2ab340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xf3949b78 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xf3949bb0 sleepq_switch(ca2ab340,0,c1583a69,29c,ca2ab340,...) at sleepq_switch+0x127/frame 0xf3949be4 sleepq_timedwait(c9afea84,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xf3949c08 _sleep(c9afea84,c9afea00,54,c15bdd59,7fffff6c,...) at _sleep+0x362/frame 0xf3949c68 softdep_flush(c969f8c4,f3949ce8,c157546c,402,0,...) at softdep_flush+0x192/frame 0xf3949cac fork_exit(c0ee3b30,c969f8c4,f3949ce8) at fork_exit+0x7e/frame 0xf3949cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf3949cd4 --- trap 0, eip = 0, esp = 0xf3949d20, ebp = 0 --- Tracing command bufspacedaemon pid 20 tid 100084 td 0xc937b340 sched_switch(c937b340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf13b88 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf13bc0 sleepq_switch(c937b340,0,c1583a69,29c,c937b340,...) at sleepq_switch+0x127/frame 0xedf13bf4 sleepq_timedwait(c1a39748,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf13c18 _sleep(c1a39748,c1a39500,25c,c1566ade,fffffed8,...) at _sleep+0x362/frame 0xedf13c78 bufspace_daemon(0,edf13ce8,c157546c,402,0,...) at bufspace_daemon+0x1ff/frame 0xedf13cac fork_exit(c0cd72a0,0,edf13ce8) at fork_exit+0x7e/frame 0xedf13cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf13cd4 --- trap 0, eip = 0, esp = 0xedf13d20, ebp = 0 --- Tracing command pagezero pid 19 tid 100083 td 0xc937b680 sched_switch(c937b680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf10b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedf10bbc sleepq_switch(c937b680,0,c1583a69,29c,c937b680,...) at sleepq_switch+0x127/frame 0xedf10bf0 sleepq_timedwait(c1a3f2a8,0,12b,0,0,...) at sleepq_timedwait+0x48/frame 0xedf10c14 _sleep(c1a3f2a8,c1a9c080,0,c15c79d5,fffea520,...) at _sleep+0x362/frame 0xedf10c74 vm_pagezero(0,edf10ce8,c157546c,402,0,...) at vm_pagezero+0xc2/frame 0xedf10cac fork_exit(c0f47e40,0,edf10ce8) at fork_exit+0x7e/frame 0xedf10cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf10cd4 --- trap 0, eip = 0, esp = 0xedf10d20, ebp = 0 --- Tracing command vmdaemon pid 18 tid 100082 td 0xc937b9c0 sched_switch(c937b9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf0db48 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf0db80 sleepq_switch(c937b9c0,0,c1583a69,279,c937b9c0,...) at sleepq_switch+0x127/frame 0xedf0dbb4 sleepq_wait(c1a3f124,74,c158d87a,0,0,...) at sleepq_wait+0x48/frame 0xedf0dbd8 _sleep(c1a3f124,c1a3f10c,74,c158d87a,0,...) at _sleep+0x3b3/frame 0xedf0dc38 vm_daemon(0,edf0dce8,c157546c,402,0,...) at vm_daemon+0xb6/frame 0xedf0dcac fork_exit(c0f3fae0,0,edf0dce8) at fork_exit+0x7e/frame 0xedf0dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf0dcd4 --- trap 0, eip = 0, esp = 0xedf0dd20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100081 td 0xc937c000 sched_switch(c937c000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf0ab00 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf0ab38 sleepq_switch(c937c000,0,c1583a69,29c,c937c000,...) at sleepq_switch+0x127/frame 0xedf0ab6c sleepq_timedwait(c1a9c104,54,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedf0ab90 _sleep(c1a9c104,c1a9c080,54,c158d87a,fffffed8,...) at _sleep+0x362/frame 0xedf0abf0 vm_pageout(0,edf0ace8,c157546c,402,0,...) at vm_pageout+0x287/frame 0xedf0acac fork_exit(c0f3e060,0,edf0ace8) at fork_exit+0x7e/frame 0xedf0acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf0acd4 --- trap 0, eip = 0, esp = 0xedf0ad20, ebp = 0 --- Tracing command pagedaemon pid 17 tid 100088 td 0xc953b9c0 sched_switch(c953b9c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf1fb90 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf1fbc8 sleepq_switch(c953b9c0,0,c1583a69,279,c953b9c0,...) at sleepq_switch+0x127/frame 0xedf1fbfc sleepq_wait(c1a3ed20,54,edf1fc6c,0,0,...) at sleepq_wait+0x48/frame 0xedf1fc20 _sleep(c1a3ed20,c1a3ed0c,54,c15c26fa,0,...) at _sleep+0x3b3/frame 0xedf1fc80 uma_reclaim_worker(0,edf1fce8,c157546c,402,0,...) at uma_reclaim_worker+0x90/frame 0xedf1fcac fork_exit(c0f1fc20,0,edf1fce8) at fork_exit+0x7e/frame 0xedf1fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf1fcd4 --- trap 0, eip = 0, esp = 0xedf1fd20, ebp = 0 --- Tracing command soaiod4 pid 16 tid 100080 td 0xc937c340 sched_switch(c937c340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf07b80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf07bb8 sleepq_switch(c937c340,0,c1583a69,29c,c937c340,...) at sleepq_switch+0x127/frame 0xedf07bec sleepq_timedwait(c1a3859c,0,1d,0,0,...) at sleepq_timedwait+0x48/frame 0xedf07c10 _sleep(c1a3859c,c1a38570,0,c1566ade,ffffdd50,...) at _sleep+0x362/frame 0xedf07c70 soaio_kproc_loop(4,edf07ce8,c157546c,402,0,...) at soaio_kproc_loop+0x13a/frame 0xedf07cac fork_exit(c0ca0470,4,edf07ce8) at fork_exit+0x7e/frame 0xedf07cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf07cd4 --- trap 0, eip = 0, esp = 0xedf07d20, ebp = 0 --- Tracing command soaiod3 pid 15 tid 100079 td 0xc937c680 sched_switch(c937c680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf04b80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf04bb8 sleepq_switch(c937c680,0,c1583a69,29c,c937c680,...) at sleepq_switch+0x127/frame 0xedf04bec sleepq_timedwait(c1a3859c,0,1d,0,0,...) at sleepq_timedwait+0x48/frame 0xedf04c10 _sleep(c1a3859c,c1a38570,0,c1566ade,ffffdd50,...) at _sleep+0x362/frame 0xedf04c70 soaio_kproc_loop(3,edf04ce8,c157546c,402,0,...) at soaio_kproc_loop+0x13a/frame 0xedf04cac fork_exit(c0ca0470,3,edf04ce8) at fork_exit+0x7e/frame 0xedf04cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf04cd4 --- trap 0, eip = 0, esp = 0xedf04d20, ebp = 0 --- Tracing command soaiod2 pid 9 tid 100078 td 0xc90b3340 sched_switch(c90b3340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedf01b80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedf01bb8 sleepq_switch(c90b3340,0,c1583a69,29c,c90b3340,...) at sleepq_switch+0x127/frame 0xedf01bec sleepq_timedwait(c1a3859c,0,1d,0,0,...) at sleepq_timedwait+0x48/frame 0xedf01c10 _sleep(c1a3859c,c1a38570,0,c1566ade,ffffdd50,...) at _sleep+0x362/frame 0xedf01c70 soaio_kproc_loop(2,edf01ce8,c157546c,402,0,...) at soaio_kproc_loop+0x13a/frame 0xedf01cac fork_exit(c0ca0470,2,edf01ce8) at fork_exit+0x7e/frame 0xedf01cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedf01cd4 --- trap 0, eip = 0, esp = 0xedf01d20, ebp = 0 --- Tracing command soaiod1 pid 8 tid 100077 td 0xc90b3680 sched_switch(c90b3680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedefeb80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedefebb8 sleepq_switch(c90b3680,0,c1583a69,29c,c90b3680,...) at sleepq_switch+0x127/frame 0xedefebec sleepq_timedwait(c1a3859c,0,1d,0,0,...) at sleepq_timedwait+0x48/frame 0xedefec10 _sleep(c1a3859c,c1a38570,0,c1566ade,ffffdd50,...) at _sleep+0x362/frame 0xedefec70 soaio_kproc_loop(1,edefece8,c157546c,402,0,...) at soaio_kproc_loop+0x13a/frame 0xedefecac fork_exit(c0ca0470,1,edefece8) at fork_exit+0x7e/frame 0xedefecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedefecd4 --- trap 0, eip = 0, esp = 0xedefed20, ebp = 0 --- Tracing command sctp_iterator pid 7 tid 100074 td 0xc90e1340 sched_switch(c90e1340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd75b90 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedd75bc8 sleepq_switch(c90e1340,0,c1583a69,279,c90e1340,...) at sleepq_switch+0x127/frame 0xedd75bfc sleepq_wait(c1a97a54,0,c15a4bc9,0,0,...) at sleepq_wait+0x48/frame 0xedd75c20 _sleep(c1a97a54,c1a97a1c,0,c15a4bc9,0,...) at _sleep+0x3b3/frame 0xedd75c80 sctp_iterator_thread(0,edd75ce8,c157546c,402,0,...) at sctp_iterator_thread+0x7c/frame 0xedd75cac fork_exit(c0da1090,0,edd75ce8) at fork_exit+0x7e/frame 0xedd75cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd75cd4 --- trap 0, eip = 0, esp = 0xedd75d20, ebp = 0 --- Tracing command fdc0 pid 6 tid 100070 td 0xc90e2340 sched_switch(c90e2340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd68b3c mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd68b74 sleepq_switch(c90e2340,0,c1583a69,29c,c90e2340,...) at sleepq_switch+0x127/frame 0xedd68ba8 sleepq_timedwait(c9055e3c,5c,0,0,0,...) at sleepq_timedwait+0x48/frame 0xedd68bcc _sleep(c9055e3c,c9055ef0,5c,c1566ade,fffffed8,...) at _sleep+0x362/frame 0xedd68c2c fdc_thread(c9055e00,edd68ce8,c157546c,402,0,...) at fdc_thread+0xb67/frame 0xedd68cac fork_exit(c110ea40,c9055e00,edd68ce8) at fork_exit+0x7e/frame 0xedd68cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd68cd4 --- trap 0, eip = 0, esp = 0xedd68d20, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc9002680 sched_switch(c9002680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedce1bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedce1be4 sleepq_switch(c9002680,0,c1583a69,279,c904aacc,...) at sleepq_switch+0x127/frame 0xedce1c18 sleepq_wait(c904aacc,0,edce1c70,1,0,...) at sleepq_wait+0x48/frame 0xedce1c3c _cv_wait(c904aacc,c904acf8,c154bb33,76,c904aad4,...) at _cv_wait+0x21b/frame 0xedce1c84 usb_process(c904aac4,edce1ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedce1cac fork_exit(c0a7cc70,c904aac4,edce1ce8) at fork_exit+0x7e/frame 0xedce1cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedce1cd4 --- trap 0, eip = 0, esp = 0xedce1d20, ebp = 0 --- Tracing command usb pid 14 tid 100036 td 0xc9002340 sched_switch(c9002340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedce4bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedce4be4 sleepq_switch(c9002340,0,c1583a69,279,c904aafc,...) at sleepq_switch+0x127/frame 0xedce4c18 sleepq_wait(c904aafc,0,edce4c70,1,0,...) at sleepq_wait+0x48/frame 0xedce4c3c _cv_wait(c904aafc,c904acf8,c154bb33,76,c904ab04,...) at _cv_wait+0x21b/frame 0xedce4c84 usb_process(c904aaf4,edce4ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedce4cac fork_exit(c0a7cc70,c904aaf4,edce4ce8) at fork_exit+0x7e/frame 0xedce4cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedce4cd4 --- trap 0, eip = 0, esp = 0xedce4d20, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc9002000 sched_switch(c9002000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedce7bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedce7be4 sleepq_switch(c9002000,0,c1583a69,279,c904ab2c,...) at sleepq_switch+0x127/frame 0xedce7c18 sleepq_wait(c904ab2c,0,edce7c70,1,0,...) at sleepq_wait+0x48/frame 0xedce7c3c _cv_wait(c904ab2c,c904acf8,c154bb33,76,c904ab34,...) at _cv_wait+0x21b/frame 0xedce7c84 usb_process(c904ab24,edce7ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedce7cac fork_exit(c0a7cc70,c904ab24,edce7ce8) at fork_exit+0x7e/frame 0xedce7cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedce7cd4 --- trap 0, eip = 0, esp = 0xedce7d20, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc8f129c0 sched_switch(c8f129c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedceabac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedceabe4 sleepq_switch(c8f129c0,0,c1583a69,279,c904ab5c,...) at sleepq_switch+0x127/frame 0xedceac18 sleepq_wait(c904ab5c,0,edceac70,1,0,...) at sleepq_wait+0x48/frame 0xedceac3c _cv_wait(c904ab5c,c904acf8,c154bb33,76,c904ab64,...) at _cv_wait+0x21b/frame 0xedceac84 usb_process(c904ab54,edceace8,c157546c,402,0,...) at usb_process+0x140/frame 0xedceacac fork_exit(c0a7cc70,c904ab54,edceace8) at fork_exit+0x7e/frame 0xedceacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedceacd4 --- trap 0, eip = 0, esp = 0xedcead20, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc8f12680 sched_switch(c8f12680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcedbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcedbe4 sleepq_switch(c8f12680,0,c1583a69,279,c904ab8c,...) at sleepq_switch+0x127/frame 0xedcedc18 sleepq_wait(c904ab8c,0,edcedc70,1,0,...) at sleepq_wait+0x48/frame 0xedcedc3c _cv_wait(c904ab8c,c904acf8,c154bb33,76,c904ab94,...) at _cv_wait+0x21b/frame 0xedcedc84 usb_process(c904ab84,edcedce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcedcac fork_exit(c0a7cc70,c904ab84,edcedce8) at fork_exit+0x7e/frame 0xedcedcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcedcd4 --- trap 0, eip = 0, esp = 0xedcedd20, ebp = 0 --- Tracing command usb pid 14 tid 100041 td 0xc90a7680 sched_switch(c90a7680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcf3bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcf3be4 sleepq_switch(c90a7680,0,c1583a69,279,c909facc,...) at sleepq_switch+0x127/frame 0xedcf3c18 sleepq_wait(c909facc,0,edcf3c70,1,0,...) at sleepq_wait+0x48/frame 0xedcf3c3c _cv_wait(c909facc,c909fcf8,c154bb33,76,c909fad4,...) at _cv_wait+0x21b/frame 0xedcf3c84 usb_process(c909fac4,edcf3ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcf3cac fork_exit(c0a7cc70,c909fac4,edcf3ce8) at fork_exit+0x7e/frame 0xedcf3cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcf3cd4 --- trap 0, eip = 0, esp = 0xedcf3d20, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc90a7340 sched_switch(c90a7340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcf6bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcf6be4 sleepq_switch(c90a7340,0,c1583a69,279,c909fafc,...) at sleepq_switch+0x127/frame 0xedcf6c18 sleepq_wait(c909fafc,0,edcf6c70,1,0,...) at sleepq_wait+0x48/frame 0xedcf6c3c _cv_wait(c909fafc,c909fcf8,c154bb33,76,c909fb04,...) at _cv_wait+0x21b/frame 0xedcf6c84 usb_process(c909faf4,edcf6ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcf6cac fork_exit(c0a7cc70,c909faf4,edcf6ce8) at fork_exit+0x7e/frame 0xedcf6cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcf6cd4 --- trap 0, eip = 0, esp = 0xedcf6d20, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc90a7000 sched_switch(c90a7000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcf9bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcf9be4 sleepq_switch(c90a7000,0,c1583a69,279,c909fb2c,...) at sleepq_switch+0x127/frame 0xedcf9c18 sleepq_wait(c909fb2c,0,edcf9c70,1,0,...) at sleepq_wait+0x48/frame 0xedcf9c3c _cv_wait(c909fb2c,c909fcf8,c154bb33,76,c909fb34,...) at _cv_wait+0x21b/frame 0xedcf9c84 usb_process(c909fb24,edcf9ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcf9cac fork_exit(c0a7cc70,c909fb24,edcf9ce8) at fork_exit+0x7e/frame 0xedcf9cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcf9cd4 --- trap 0, eip = 0, esp = 0xedcf9d20, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc90a69c0 sched_switch(c90a69c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcfcbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcfcbe4 sleepq_switch(c90a69c0,0,c1583a69,279,c909fb5c,...) at sleepq_switch+0x127/frame 0xedcfcc18 sleepq_wait(c909fb5c,0,edcfcc70,1,0,...) at sleepq_wait+0x48/frame 0xedcfcc3c _cv_wait(c909fb5c,c909fcf8,c154bb33,76,c909fb64,...) at _cv_wait+0x21b/frame 0xedcfcc84 usb_process(c909fb54,edcfcce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcfccac fork_exit(c0a7cc70,c909fb54,edcfcce8) at fork_exit+0x7e/frame 0xedcfccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcfccd4 --- trap 0, eip = 0, esp = 0xedcfcd20, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc90a6680 sched_switch(c90a6680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedcffbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedcffbe4 sleepq_switch(c90a6680,0,c1583a69,279,c909fb8c,...) at sleepq_switch+0x127/frame 0xedcffc18 sleepq_wait(c909fb8c,0,edcffc70,1,0,...) at sleepq_wait+0x48/frame 0xedcffc3c _cv_wait(c909fb8c,c909fcf8,c154bb33,76,c909fb94,...) at _cv_wait+0x21b/frame 0xedcffc84 usb_process(c909fb84,edcffce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedcffcac fork_exit(c0a7cc70,c909fb84,edcffce8) at fork_exit+0x7e/frame 0xedcffcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcffcd4 --- trap 0, eip = 0, esp = 0xedcffd20, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc90a6000 sched_switch(c90a6000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd05bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd05be4 sleepq_switch(c90a6000,0,c1583a69,279,c90a8acc,...) at sleepq_switch+0x127/frame 0xedd05c18 sleepq_wait(c90a8acc,0,edd05c70,1,0,...) at sleepq_wait+0x48/frame 0xedd05c3c _cv_wait(c90a8acc,c90a8cf8,c154bb33,76,c90a8ad4,...) at _cv_wait+0x21b/frame 0xedd05c84 usb_process(c90a8ac4,edd05ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd05cac fork_exit(c0a7cc70,c90a8ac4,edd05ce8) at fork_exit+0x7e/frame 0xedd05cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd05cd4 --- trap 0, eip = 0, esp = 0xedd05d20, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc90a59c0 sched_switch(c90a59c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd08bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd08be4 sleepq_switch(c90a59c0,0,c1583a69,279,c90a8afc,...) at sleepq_switch+0x127/frame 0xedd08c18 sleepq_wait(c90a8afc,0,edd08c70,1,0,...) at sleepq_wait+0x48/frame 0xedd08c3c _cv_wait(c90a8afc,c90a8cf8,c154bb33,76,c90a8b04,...) at _cv_wait+0x21b/frame 0xedd08c84 usb_process(c90a8af4,edd08ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd08cac fork_exit(c0a7cc70,c90a8af4,edd08ce8) at fork_exit+0x7e/frame 0xedd08cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd08cd4 --- trap 0, eip = 0, esp = 0xedd08d20, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc90a5680 sched_switch(c90a5680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd0bbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd0bbe4 sleepq_switch(c90a5680,0,c1583a69,279,c90a8b2c,...) at sleepq_switch+0x127/frame 0xedd0bc18 sleepq_wait(c90a8b2c,0,edd0bc70,1,0,...) at sleepq_wait+0x48/frame 0xedd0bc3c _cv_wait(c90a8b2c,c90a8cf8,c154bb33,76,c90a8b34,...) at _cv_wait+0x21b/frame 0xedd0bc84 usb_process(c90a8b24,edd0bce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd0bcac fork_exit(c0a7cc70,c90a8b24,edd0bce8) at fork_exit+0x7e/frame 0xedd0bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd0bcd4 --- trap 0, eip = 0, esp = 0xedd0bd20, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc90a5340 sched_switch(c90a5340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd0ebac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd0ebe4 sleepq_switch(c90a5340,0,c1583a69,279,c90a8b5c,...) at sleepq_switch+0x127/frame 0xedd0ec18 sleepq_wait(c90a8b5c,0,edd0ec70,1,0,...) at sleepq_wait+0x48/frame 0xedd0ec3c _cv_wait(c90a8b5c,c90a8cf8,c154bb33,76,c90a8b64,...) at _cv_wait+0x21b/frame 0xedd0ec84 usb_process(c90a8b54,edd0ece8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd0ecac fork_exit(c0a7cc70,c90a8b54,edd0ece8) at fork_exit+0x7e/frame 0xedd0ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd0ecd4 --- trap 0, eip = 0, esp = 0xedd0ed20, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc90a5000 sched_switch(c90a5000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd11bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd11be4 sleepq_switch(c90a5000,0,c1583a69,279,c90a8b8c,...) at sleepq_switch+0x127/frame 0xedd11c18 sleepq_wait(c90a8b8c,0,edd11c70,1,0,...) at sleepq_wait+0x48/frame 0xedd11c3c _cv_wait(c90a8b8c,c90a8cf8,c154bb33,76,c90a8b94,...) at _cv_wait+0x21b/frame 0xedd11c84 usb_process(c90a8b84,edd11ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd11cac fork_exit(c0a7cc70,c90a8b84,edd11ce8) at fork_exit+0x7e/frame 0xedd11cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd11cd4 --- trap 0, eip = 0, esp = 0xedd11d20, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc90049c0 sched_switch(c90049c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd15bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd15be4 sleepq_switch(c90049c0,0,c1583a69,279,c90aeacc,...) at sleepq_switch+0x127/frame 0xedd15c18 sleepq_wait(c90aeacc,0,edd15c70,1,0,...) at sleepq_wait+0x48/frame 0xedd15c3c _cv_wait(c90aeacc,c90aecf8,c154bb33,76,c90aead4,...) at _cv_wait+0x21b/frame 0xedd15c84 usb_process(c90aeac4,edd15ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd15cac fork_exit(c0a7cc70,c90aeac4,edd15ce8) at fork_exit+0x7e/frame 0xedd15cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd15cd4 --- trap 0, eip = 0, esp = 0xedd15d20, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xc90b3000 sched_switch(c90b3000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd18bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd18be4 sleepq_switch(c90b3000,0,c1583a69,279,c90aeafc,...) at sleepq_switch+0x127/frame 0xedd18c18 sleepq_wait(c90aeafc,0,edd18c70,1,0,...) at sleepq_wait+0x48/frame 0xedd18c3c _cv_wait(c90aeafc,c90aecf8,c154bb33,76,c90aeb04,...) at _cv_wait+0x21b/frame 0xedd18c84 usb_process(c90aeaf4,edd18ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd18cac fork_exit(c0a7cc70,c90aeaf4,edd18ce8) at fork_exit+0x7e/frame 0xedd18cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd18cd4 --- trap 0, eip = 0, esp = 0xedd18d20, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc90b29c0 sched_switch(c90b29c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd1bbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd1bbe4 sleepq_switch(c90b29c0,0,c1583a69,279,c90aeb2c,...) at sleepq_switch+0x127/frame 0xedd1bc18 sleepq_wait(c90aeb2c,0,edd1bc70,1,0,...) at sleepq_wait+0x48/frame 0xedd1bc3c _cv_wait(c90aeb2c,c90aecf8,c154bb33,76,c90aeb34,...) at _cv_wait+0x21b/frame 0xedd1bc84 usb_process(c90aeb24,edd1bce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd1bcac fork_exit(c0a7cc70,c90aeb24,edd1bce8) at fork_exit+0x7e/frame 0xedd1bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd1bcd4 --- trap 0, eip = 0, esp = 0xedd1bd20, ebp = 0 --- Tracing command usb pid 14 tid 100055 td 0xc90b2680 sched_switch(c90b2680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd1ebac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd1ebe4 sleepq_switch(c90b2680,0,c1583a69,279,c90aeb5c,...) at sleepq_switch+0x127/frame 0xedd1ec18 sleepq_wait(c90aeb5c,0,edd1ec70,1,0,...) at sleepq_wait+0x48/frame 0xedd1ec3c _cv_wait(c90aeb5c,c90aecf8,c154bb33,76,c90aeb64,...) at _cv_wait+0x21b/frame 0xedd1ec84 usb_process(c90aeb54,edd1ece8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd1ecac fork_exit(c0a7cc70,c90aeb54,edd1ece8) at fork_exit+0x7e/frame 0xedd1ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd1ecd4 --- trap 0, eip = 0, esp = 0xedd1ed20, ebp = 0 --- Tracing command usb pid 14 tid 100056 td 0xc90b2340 sched_switch(c90b2340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd21bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd21be4 sleepq_switch(c90b2340,0,c1583a69,279,c90aeb8c,...) at sleepq_switch+0x127/frame 0xedd21c18 sleepq_wait(c90aeb8c,0,edd21c70,1,0,...) at sleepq_wait+0x48/frame 0xedd21c3c _cv_wait(c90aeb8c,c90aecf8,c154bb33,76,c90aeb94,...) at _cv_wait+0x21b/frame 0xedd21c84 usb_process(c90aeb84,edd21ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd21cac fork_exit(c0a7cc70,c90aeb84,edd21ce8) at fork_exit+0x7e/frame 0xedd21cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd21cd4 --- trap 0, eip = 0, esp = 0xedd21d20, ebp = 0 --- Tracing command usb pid 14 tid 100057 td 0xc90b2000 sched_switch(c90b2000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd25bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd25be4 sleepq_switch(c90b2000,0,c1583a69,279,c90b8acc,...) at sleepq_switch+0x127/frame 0xedd25c18 sleepq_wait(c90b8acc,0,edd25c70,1,0,...) at sleepq_wait+0x48/frame 0xedd25c3c _cv_wait(c90b8acc,c90b8cf8,c154bb33,76,c90b8ad4,...) at _cv_wait+0x21b/frame 0xedd25c84 usb_process(c90b8ac4,edd25ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd25cac fork_exit(c0a7cc70,c90b8ac4,edd25ce8) at fork_exit+0x7e/frame 0xedd25cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd25cd4 --- trap 0, eip = 0, esp = 0xedd25d20, ebp = 0 --- Tracing command usb pid 14 tid 100058 td 0xc90b19c0 sched_switch(c90b19c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd28bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd28be4 sleepq_switch(c90b19c0,0,c1583a69,279,c90b8afc,...) at sleepq_switch+0x127/frame 0xedd28c18 sleepq_wait(c90b8afc,0,edd28c70,1,0,...) at sleepq_wait+0x48/frame 0xedd28c3c _cv_wait(c90b8afc,c90b8cf8,c154bb33,76,c90b8b04,...) at _cv_wait+0x21b/frame 0xedd28c84 usb_process(c90b8af4,edd28ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd28cac fork_exit(c0a7cc70,c90b8af4,edd28ce8) at fork_exit+0x7e/frame 0xedd28cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd28cd4 --- trap 0, eip = 0, esp = 0xedd28d20, ebp = 0 --- Tracing command usb pid 14 tid 100059 td 0xc90b1680 sched_switch(c90b1680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd2bbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd2bbe4 sleepq_switch(c90b1680,0,c1583a69,279,c90b8b2c,...) at sleepq_switch+0x127/frame 0xedd2bc18 sleepq_wait(c90b8b2c,0,edd2bc70,1,0,...) at sleepq_wait+0x48/frame 0xedd2bc3c _cv_wait(c90b8b2c,c90b8cf8,c154bb33,76,c90b8b34,...) at _cv_wait+0x21b/frame 0xedd2bc84 usb_process(c90b8b24,edd2bce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd2bcac fork_exit(c0a7cc70,c90b8b24,edd2bce8) at fork_exit+0x7e/frame 0xedd2bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2bcd4 --- trap 0, eip = 0, esp = 0xedd2bd20, ebp = 0 --- Tracing command usb pid 14 tid 100060 td 0xc90b1340 sched_switch(c90b1340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd2ebac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd2ebe4 sleepq_switch(c90b1340,0,c1583a69,279,c90b8b5c,...) at sleepq_switch+0x127/frame 0xedd2ec18 sleepq_wait(c90b8b5c,0,edd2ec70,1,0,...) at sleepq_wait+0x48/frame 0xedd2ec3c _cv_wait(c90b8b5c,c90b8cf8,c154bb33,76,c90b8b64,...) at _cv_wait+0x21b/frame 0xedd2ec84 usb_process(c90b8b54,edd2ece8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd2ecac fork_exit(c0a7cc70,c90b8b54,edd2ece8) at fork_exit+0x7e/frame 0xedd2ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd2ecd4 --- trap 0, eip = 0, esp = 0xedd2ed20, ebp = 0 --- Tracing command usb pid 14 tid 100061 td 0xc90b1000 sched_switch(c90b1000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd31bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd31be4 sleepq_switch(c90b1000,0,c1583a69,279,c90b8b8c,...) at sleepq_switch+0x127/frame 0xedd31c18 sleepq_wait(c90b8b8c,0,edd31c70,1,0,...) at sleepq_wait+0x48/frame 0xedd31c3c _cv_wait(c90b8b8c,c90b8cf8,c154bb33,76,c90b8b94,...) at _cv_wait+0x21b/frame 0xedd31c84 usb_process(c90b8b84,edd31ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd31cac fork_exit(c0a7cc70,c90b8b84,edd31ce8) at fork_exit+0x7e/frame 0xedd31cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd31cd4 --- trap 0, eip = 0, esp = 0xedd31d20, ebp = 0 --- Tracing command usb pid 14 tid 100063 td 0xc90b0680 sched_switch(c90b0680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd39bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd39be4 sleepq_switch(c90b0680,0,c1583a69,279,c904fca4,...) at sleepq_switch+0x127/frame 0xedd39c18 sleepq_wait(c904fca4,0,edd39c70,1,0,...) at sleepq_wait+0x48/frame 0xedd39c3c _cv_wait(c904fca4,c904fed0,c154bb33,76,c904fcac,...) at _cv_wait+0x21b/frame 0xedd39c84 usb_process(c904fc9c,edd39ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd39cac fork_exit(c0a7cc70,c904fc9c,edd39ce8) at fork_exit+0x7e/frame 0xedd39cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd39cd4 --- trap 0, eip = 0, esp = 0xedd39d20, ebp = 0 --- Tracing command usb pid 14 tid 100064 td 0xc90b0340 sched_switch(c90b0340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd3cbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd3cbe4 sleepq_switch(c90b0340,0,c1583a69,279,c904fcd4,...) at sleepq_switch+0x127/frame 0xedd3cc18 sleepq_wait(c904fcd4,0,edd3cc70,1,0,...) at sleepq_wait+0x48/frame 0xedd3cc3c _cv_wait(c904fcd4,c904fed0,c154bb33,76,c904fcdc,...) at _cv_wait+0x21b/frame 0xedd3cc84 usb_process(c904fccc,edd3cce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd3ccac fork_exit(c0a7cc70,c904fccc,edd3cce8) at fork_exit+0x7e/frame 0xedd3ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd3ccd4 --- trap 0, eip = 0, esp = 0xedd3cd20, ebp = 0 --- Tracing command usb pid 14 tid 100065 td 0xc90b0000 sched_switch(c90b0000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd3fbac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd3fbe4 sleepq_switch(c90b0000,0,c1583a69,279,c904fd04,...) at sleepq_switch+0x127/frame 0xedd3fc18 sleepq_wait(c904fd04,0,edd3fc70,1,0,...) at sleepq_wait+0x48/frame 0xedd3fc3c _cv_wait(c904fd04,c904fed0,c154bb33,76,c904fd0c,...) at _cv_wait+0x21b/frame 0xedd3fc84 usb_process(c904fcfc,edd3fce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd3fcac fork_exit(c0a7cc70,c904fcfc,edd3fce8) at fork_exit+0x7e/frame 0xedd3fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd3fcd4 --- trap 0, eip = 0, esp = 0xedd3fd20, ebp = 0 --- Tracing command usb pid 14 tid 100066 td 0xc90e3340 sched_switch(c90e3340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd42bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd42be4 sleepq_switch(c90e3340,0,c1583a69,279,c904fd34,...) at sleepq_switch+0x127/frame 0xedd42c18 sleepq_wait(c904fd34,0,edd42c70,1,0,...) at sleepq_wait+0x48/frame 0xedd42c3c _cv_wait(c904fd34,c904fed0,c154bb33,76,c904fd3c,...) at _cv_wait+0x21b/frame 0xedd42c84 usb_process(c904fd2c,edd42ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd42cac fork_exit(c0a7cc70,c904fd2c,edd42ce8) at fork_exit+0x7e/frame 0xedd42cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd42cd4 --- trap 0, eip = 0, esp = 0xedd42d20, ebp = 0 --- Tracing command usb pid 14 tid 100067 td 0xc90e3000 sched_switch(c90e3000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd45bac mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd45be4 sleepq_switch(c90e3000,0,c1583a69,279,c904fd64,...) at sleepq_switch+0x127/frame 0xedd45c18 sleepq_wait(c904fd64,0,edd45c70,1,0,...) at sleepq_wait+0x48/frame 0xedd45c3c _cv_wait(c904fd64,c904fed0,c154bb33,76,c904fd6c,...) at _cv_wait+0x21b/frame 0xedd45c84 usb_process(c904fd5c,edd45ce8,c157546c,402,0,...) at usb_process+0x140/frame 0xedd45cac fork_exit(c0a7cc70,c904fd5c,edd45ce8) at fork_exit+0x7e/frame 0xedd45cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd45cd4 --- trap 0, eip = 0, esp = 0xedd45d20, ebp = 0 --- Tracing command cam pid 5 tid 100021 td 0xc8f10680 sched_switch(c8f10680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b65b68 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b65ba0 sleepq_switch(c8f10680,0,c1583a69,279,c8f10680,...) at sleepq_switch+0x127/frame 0xe4b65bd4 sleepq_wait(c1861f00,5c,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b65bf8 _sleep(c1861f00,c1861e80,5c,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b65c58 xpt_done_td(c1861e80,e4b65ce8,c157546c,402,0,...) at xpt_done_td+0x17e/frame 0xe4b65cac fork_exit(c04d9e70,c1861e80,e4b65ce8) at fork_exit+0x7e/frame 0xe4b65cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b65cd4 --- trap 0, eip = 0, esp = 0xe4b65d20, ebp = 0 --- Tracing command cam pid 5 tid 100076 td 0xc90b39c0 sched_switch(c90b39c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd7bb68 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedd7bba0 sleepq_switch(c90b39c0,0,c1583a69,279,c90b39c0,...) at sleepq_switch+0x127/frame 0xedd7bbd4 sleepq_wait(c1861dac,5c,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xedd7bbf8 _sleep(c1861dac,c1861dfc,5c,c1566ade,0,...) at _sleep+0x3b3/frame 0xedd7bc58 xpt_scanner_thread(0,edd7bce8,c157546c,402,0,...) at xpt_scanner_thread+0xa2/frame 0xedd7bcac fork_exit(c04da220,0,edd7bce8) at fork_exit+0x7e/frame 0xedd7bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd7bcd4 --- trap 0, eip = 0, esp = 0xedd7bd20, ebp = 0 --- Tracing command crypto returns pid 4 tid 100019 td 0xc8f11000 sched_switch(c8f11000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b5fb70 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b5fba8 sleepq_switch(c8f11000,0,c1583a69,279,c8f11000,...) at sleepq_switch+0x127/frame 0xe4b5fbdc sleepq_wait(c1a3e078,6c,c15b5c34,0,0,...) at sleepq_wait+0x48/frame 0xe4b5fc00 _sleep(c1a3e078,c1a3e064,6c,c15b5c34,0,...) at _sleep+0x3b3/frame 0xe4b5fc60 crypto_ret_proc(0,e4b5fce8,c157546c,402,0,...) at crypto_ret_proc+0x1f5/frame 0xe4b5fcac fork_exit(c0e8aa80,0,e4b5fce8) at fork_exit+0x7e/frame 0xe4b5fcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5fcd4 --- trap 0, eip = 0, esp = 0xe4b5fd20, ebp = 0 --- Tracing command crypto pid 3 tid 100018 td 0xc8f11340 sched_switch(c8f11340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b5cb80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b5cbb8 sleepq_switch(c8f11340,0,c1583a69,279,c8f11340,...) at sleepq_switch+0x127/frame 0xe4b5cbec sleepq_wait(c1a3dfc0,6c,c15b5c28,0,0,...) at sleepq_wait+0x48/frame 0xe4b5cc10 _sleep(c1a3dfc0,c1a3dfa8,6c,c15b5c28,0,...) at _sleep+0x3b3/frame 0xe4b5cc70 crypto_proc(0,e4b5cce8,c157546c,402,0,...) at crypto_proc+0x2c1/frame 0xe4b5ccac fork_exit(c0e8a650,0,e4b5cce8) at fork_exit+0x7e/frame 0xe4b5ccd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b5ccd4 --- trap 0, eip = 0, esp = 0xe4b5cd20, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc8f12000 sched_switch(c8f12000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b53b80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b53bb8 sleepq_switch(c8f12000,0,c1583a69,279,c8f12000,...) at sleepq_switch+0x127/frame 0xe4b53bec sleepq_wait(c1a92bec,5c,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b53c10 _sleep(c1a92bec,c19bc5a4,25c,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b53c70 g_run_events(0,e4b53ce8,c157546c,402,0,...) at g_run_events+0x67/frame 0xe4b53cac fork_exit(c0b79230,0,e4b53ce8) at fork_exit+0x7e/frame 0xe4b53cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b53cd4 --- trap 0, eip = 0, esp = 0xe4b53d20, ebp = 0 --- Tracing command geom pid 13 tid 100016 td 0xc8f119c0 sched_switch(c8f119c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b56b70 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b56ba8 sleepq_switch(c8f119c0,0,c1583a69,279,c8f119c0,...) at sleepq_switch+0x127/frame 0xe4b56bdc sleepq_wait(c1a92bf0,5c,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b56c00 _sleep(c1a92bf0,c19bc5ec,25c,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b56c60 g_io_schedule_up(c8f119c0,0,c156890d,60,e4b56cd4,...) at g_io_schedule_up+0x12a/frame 0xe4b56c94 g_up_procbody(0,e4b56ce8,c157546c,402,0,...) at g_up_procbody+0x9d/frame 0xe4b56cac fork_exit(c0b792d0,0,e4b56ce8) at fork_exit+0x7e/frame 0xe4b56cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b56cd4 --- trap 0, eip = 0, esp = 0xe4b56d20, ebp = 0 --- Tracing command geom pid 13 tid 100017 td 0xc8f11680 sched_switch(c8f11680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b59b6c mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b59ba4 sleepq_switch(c8f11680,0,c1583a69,279,c8f11680,...) at sleepq_switch+0x127/frame 0xe4b59bd8 sleepq_wait(c1a92bf4,5c,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b59bfc _sleep(c1a92bf4,c19bc5cc,25c,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b59c5c g_io_schedule_down(c8f11680,0,c156890d,6d,e4b59cd4,...) at g_io_schedule_down+0xec/frame 0xe4b59c94 g_down_procbody(0,e4b59ce8,c157546c,402,0,...) at g_down_procbody+0x9d/frame 0xe4b59cac fork_exit(c0b79370,0,e4b59ce8) at fork_exit+0x7e/frame 0xe4b59cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b59cd4 --- trap 0, eip = 0, esp = 0xe4b59d20, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc8e47340 sched_switch(c8e47340,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xe4b3ec38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xe4b3ec70 ithread_loop(c8ea4020,e4b3ece8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xe4b3ecac fork_exit(c0bf3c60,c8ea4020,e4b3ece8) at fork_exit+0x7e/frame 0xe4b3ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3ecd4 --- trap 0, eip = 0, esp = 0xe4b3ed20, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc8e47000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xc8e469c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc8e46680 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xc8e46340 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100013 td 0xc8e46000 sched_switch(c8e46000,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xe4b4dc38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xe4b4dc70 ithread_loop(c8ea3f70,e4b4dce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xe4b4dcac fork_exit(c0bf3c60,c8ea3f70,e4b4dce8) at fork_exit+0x7e/frame 0xe4b4dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b4dcd4 --- trap 0, eip = 0, esp = 0xe4b4dd20, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc8e49340 sched_switch(c8e49340,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xe4b74c38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xe4b74c70 ithread_loop(c8ea3f60,e4b74ce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xe4b74cac fork_exit(c0bf3c60,c8ea3f60,e4b74ce8) at fork_exit+0x7e/frame 0xe4b74cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b74cd4 --- trap 0, eip = 0, esp = 0xe4b74d20, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc9004340 sched_switch(c9004340,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xe4b7ac38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xe4b7ac70 ithread_loop(c8ea3f50,e4b7ace8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xe4b7acac fork_exit(c0bf3c60,c8ea3f50,e4b7ace8) at fork_exit+0x7e/frame 0xe4b7acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7acd4 --- trap 0, eip = 0, esp = 0xe4b7ad20, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc90039c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xc9003680 sched_switch(c9003680,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xe4bb2c38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xe4bb2c70 ithread_loop(c8ea3f30,e4bb2ce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xe4bb2cac fork_exit(c0bf3c60,c8ea3f30,e4bb2ce8) at fork_exit+0x7e/frame 0xe4bb2cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4bb2cd4 --- trap 0, eip = 0, esp = 0xe4bb2d20, ebp = 0 --- Tracing command intr pid 12 tid 100032 td 0xc9003340 sched_switch(c9003340,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xf12bac38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xf12bac70 ithread_loop(c8ea3f20,f12bace8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xf12bacac fork_exit(c0bf3c60,c8ea3f20,f12bace8) at fork_exit+0x7e/frame 0xf12bacd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf12bacd4 --- trap 0, eip = 0, esp = 0xf12bad20, ebp = 0 --- Tracing command intr pid 12 tid 100033 td 0xc9003000 sched_switch(c9003000,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xf12bec38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xf12bec70 ithread_loop(c8ea3f10,f12bece8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xf12becac fork_exit(c0bf3c60,c8ea3f10,f12bece8) at fork_exit+0x7e/frame 0xf12becd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf12becd4 --- trap 0, eip = 0, esp = 0xf12bed20, ebp = 0 --- Tracing command intr pid 12 tid 100034 td 0xc90029c0 sched_switch(c90029c0,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xf13fdc38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xf13fdc70 ithread_loop(c8ea3f00,f13fdce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xf13fdcac fork_exit(c0bf3c60,c8ea3f00,f13fdce8) at fork_exit+0x7e/frame 0xf13fdcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xf13fdcd4 --- trap 0, eip = 0, esp = 0xf13fdd20, ebp = 0 --- Tracing command intr pid 12 tid 100040 td 0xc90a79c0 sched_switch(c90a79c0,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xedcf0c38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xedcf0c70 ithread_loop(c8ea3ef0,edcf0ce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xedcf0cac fork_exit(c0bf3c60,c8ea3ef0,edcf0ce8) at fork_exit+0x7e/frame 0xedcf0cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedcf0cd4 --- trap 0, eip = 0, esp = 0xedcf0d20, ebp = 0 --- Tracing command intr pid 12 tid 100046 td 0xc90a6340 sched_switch(c90a6340,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xedd02c38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xedd02c70 ithread_loop(c8ea3ee0,edd02ce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xedd02cac fork_exit(c0bf3c60,c8ea3ee0,edd02ce8) at fork_exit+0x7e/frame 0xedd02cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd02cd4 --- trap 0, eip = 0, esp = 0xedd02d20, ebp = 0 --- Tracing command intr pid 12 tid 100062 td 0xc90b09c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100068 td 0xc90e29c0 sched_switch(c90e29c0,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xedd4ac38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xedd4ac70 ithread_loop(c8ea3ec0,edd4ace8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xedd4acac fork_exit(c0bf3c60,c8ea3ec0,edd4ace8) at fork_exit+0x7e/frame 0xedd4acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd4acd4 --- trap 0, eip = 0, esp = 0xedd4ad20, ebp = 0 --- Tracing command intr pid 12 tid 100069 td 0xc90e2680 sched_switch(c90e2680,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xedd5bc38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xedd5bc70 ithread_loop(c8ea3eb0,edd5bce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xedd5bcac fork_exit(c0bf3c60,c8ea3eb0,edd5bce8) at fork_exit+0x7e/frame 0xedd5bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd5bcd4 --- trap 0, eip = 0, esp = 0xedd5bd20, ebp = 0 --- Tracing command intr pid 12 tid 100071 td 0xc90e2000 sched_switch(c90e2000,0,109,1b4,c15756f1,...) at sched_switch+0x4da/frame 0xedd6bc38 mi_switch(109,0,c15756f1,555,c0c5fcdf,...) at mi_switch+0x1b4/frame 0xedd6bc70 ithread_loop(c8ea3ea0,edd6bce8,c157546c,402,0,...) at ithread_loop+0x1f0/frame 0xedd6bcac fork_exit(c0bf3c60,c8ea3ea0,edd6bce8) at fork_exit+0x7e/frame 0xedd6bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd6bcd4 --- trap 0, eip = 0, esp = 0xedd6bd20, ebp = 0 --- Tracing command intr pid 12 tid 100072 td 0xc90e19c0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100004 td 0xc8e48340 sched_switch(c8e48340,0,108,1b0,c157fdda,...) at sched_switch+0x4da/frame 0xe4b31c00 mi_switch(108,0,c157fdda,a62,c19ce180,...) at mi_switch+0x1b4/frame 0xe4b31c38 sched_idletd(0,e4b31ce8,c157546c,402,0,...) at sched_idletd+0xe8/frame 0xe4b31cac fork_exit(c0c5f6f0,0,e4b31ce8) at fork_exit+0x7e/frame 0xe4b31cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b31cd4 --- trap 0, eip = 0, esp = 0xe4b31d20, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc8e48000 sched_switch(c8e48000,0,108,1b0,c19ce180,...) at sched_switch+0x4da/frame 0xe4b34c00 mi_switch(108,0,c157fdda,40e,c19ce180,...) at mi_switch+0x1b4/frame 0xe4b34c38 sched_idletd(0,e4b34ce8,c157546c,402,0,...) at sched_idletd+0x465/frame 0xe4b34cac fork_exit(c0c5f6f0,0,e4b34ce8) at fork_exit+0x7e/frame 0xe4b34cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b34cd4 --- trap 0, eip = 0, esp = 0xe4b34d20, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc8e479c0 sched_switch(c8e479c0,0,608,1b0,d3,...) at sched_switch+0x4da/frame 0xe4b37be4 mi_switch(608,0,c157d914,d3,c19cdb08,...) at mi_switch+0x1b4/frame 0xe4b37c1c critical_exit(0,e4b37c80,c157fdda,39d,c19cd380,...) at critical_exit+0xa2/frame 0xe4b37c38 sched_idletd(0,e4b37ce8,c157546c,402,0,...) at sched_idletd+0x403/frame 0xe4b37cac fork_exit(c0c5f6f0,0,e4b37ce8) at fork_exit+0x7e/frame 0xe4b37cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b37cd4 --- trap 0, eip = 0, esp = 0xe4b37d20, ebp = 0 --- Tracing command idle pid 11 tid 100007 td 0xc8e47680 sched_switch(c8e47680,0,608,1b0,d3,...) at sched_switch+0x4da/frame 0xe4b3abe4 mi_switch(608,0,c157d914,d3,c19ce208,...) at mi_switch+0x1b4/frame 0xe4b3ac1c critical_exit(0,e4b3ac80,c157fdda,39d,c19cd380,...) at critical_exit+0xa2/frame 0xe4b3ac38 sched_idletd(0,e4b3ace8,c157546c,402,0,...) at sched_idletd+0x403/frame 0xe4b3acac fork_exit(c0c5f6f0,0,e4b3ace8) at fork_exit+0x7e/frame 0xe4b3acd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b3acd4 --- trap 0, eip = 0, esp = 0xe4b3ad20, ebp = 0 --- Tracing command rand_harvestq pid 2 tid 100003 td 0xc8e48680 sched_switch(c8e48680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b2eb60 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b2eb98 sleepq_switch(c8e48680,0,c1583a69,29c,c8e48680,...) at sleepq_switch+0x127/frame 0xe4b2ebcc sleepq_timedwait(c1915e34,0,0,0,0,...) at sleepq_timedwait+0x48/frame 0xe4b2ebf0 _sleep(c1915e34,0,0,c1566ade,19999999,...) at _sleep+0x362/frame 0xe4b2ec50 random_kthread(0,e4b2ece8,c157546c,402,0,...) at random_kthread+0x24b/frame 0xe4b2ecac fork_exit(c095bba0,0,e4b2ece8) at fork_exit+0x7e/frame 0xe4b2ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b2ecd4 --- trap 0, eip = 0, esp = 0xe4b2ed20, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc8e489c0 sched_switch(c8e489c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b2a968 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b2a9a0 sleepq_switch(c8e489c0,0,c1583a69,1b8,c19cd390,...) at sleepq_switch+0x127/frame 0xe4b2a9d4 sleepq_catch_signals(c1583a69,16f,14,c6,c8e489c0,...) at sleepq_catch_signals+0x3b7/frame 0xe4b2aa10 sleepq_wait_sig(c8e44a08,6c,c1587ae8,100,0,...) at sleepq_wait_sig+0x14/frame 0xe4b2aa34 _sleep(c8e44a08,c8e44ab0,16c,c1587ae8,0,...) at _sleep+0x37e/frame 0xe4b2aa94 kern_wait6(c8e489c0,7,0,0,e4b2ab2c,...) at kern_wait6+0x47e/frame 0xe4b2ab00 sys_wait4(c8e489c0,e4b2aca8,c1574eec,4,c119f5d9,...) at sys_wait4+0x8b/frame 0xe4b2ac20 syscall(e4b2ace8) at syscall+0x388/frame 0xe4b2acdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xe4b2acdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8069af7, esp = 0xbfbfe91c, ebp = 0xbfbfe930 --- Tracing command audit pid 10 tid 100001 td 0xc8e49000 sched_switch(c8e49000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b27b80 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b27bb8 sleepq_switch(c8e49000,0,c1583a69,279,c1a9a94c,...) at sleepq_switch+0x127/frame 0xe4b27bec sleepq_wait(c1a9a94c,0,e4b27c44,1,0,...) at sleepq_wait+0x48/frame 0xe4b27c10 _cv_wait(c1a9a94c,c1a9a8d8,c15b80d6,1ab,acd,...) at _cv_wait+0x21b/frame 0xe4b27c58 audit_worker(0,e4b27ce8,c157546c,402,0,...) at audit_worker+0x94/frame 0xe4b27cac fork_exit(c0ead1d0,0,e4b27ce8) at fork_exit+0x7e/frame 0xe4b27cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b27cd4 --- trap 0, eip = 0, esp = 0xe4b27d20, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1a92f70 sched_switch(c1a92f70,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xc2022c00 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xc2022c38 sleepq_switch(c1a92f70,0,c1583a69,29c,c1a92f70,...) at sleepq_switch+0x127/frame 0xc2022c6c sleepq_timedwait(c1a92c14,54,9,0,0,...) at sleepq_timedwait+0x48/frame 0xc2022c90 _sleep(c1a92c14,0,54,c15c3120,fffff470,...) at _sleep+0x362/frame 0xc2022cf0 swapper() at swapper+0x2c7/frame 0xc2022d38 begin() at begin+0x22 Tracing command kernel pid 0 tid 100014 td 0xc8f12340 sched_switch(c8f12340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b50b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b50bbc sleepq_switch(c8f12340,0,c1583a69,279,c8f12340,...) at sleepq_switch+0x127/frame 0xe4b50bf0 sleepq_wait(c8f0f000,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b50c14 _sleep(c8f0f000,c8f0f018,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b50c74 taskqueue_thread_loop(c19dadc8,e4b50ce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b50cac fork_exit(c0c83630,c19dadc8,e4b50ce8) at fork_exit+0x7e/frame 0xe4b50cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b50cd4 --- trap 0, eip = 0, esp = 0xe4b50d20, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc8f109c0 sched_switch(c8f109c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b62b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b62bbc sleepq_switch(c8f109c0,0,c1583a69,279,c8f109c0,...) at sleepq_switch+0x127/frame 0xe4b62bf0 sleepq_wait(c8f0ef00,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b62c14 _sleep(c8f0ef00,c8f0ef18,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b62c74 taskqueue_thread_loop(c1a97300,e4b62ce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b62cac fork_exit(c0c83630,c1a97300,e4b62ce8) at fork_exit+0x7e/frame 0xe4b62cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b62cd4 --- trap 0, eip = 0, esp = 0xe4b62d20, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc8f10340 sched_switch(c8f10340,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b68b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b68bbc sleepq_switch(c8f10340,0,c1583a69,279,c8f10340,...) at sleepq_switch+0x127/frame 0xe4b68bf0 sleepq_wait(c8f0ed80,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b68c14 _sleep(c8f0ed80,c8f0ed98,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b68c74 taskqueue_thread_loop(c1a934a0,e4b68ce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b68cac fork_exit(c0c83630,c1a934a0,e4b68ce8) at fork_exit+0x7e/frame 0xe4b68cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b68cd4 --- trap 0, eip = 0, esp = 0xe4b68d20, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc8f10000 sched_switch(c8f10000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b6bba0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b6bbd8 sleepq_switch(c8f10000,0,c1583a69,279,c8f0ed00,...) at sleepq_switch+0x127/frame 0xe4b6bc0c sleepq_wait(c8f0ed00,0,c157df6a,c1566ade,0,...) at sleepq_wait+0x48/frame 0xe4b6bc30 msleep_spin_sbt(c8f0ed00,c8f0ed18,c1566ade,0,0,...) at msleep_spin_sbt+0x22d/frame 0xe4b6bc74 taskqueue_thread_loop(c1a8a224,e4b6bce8,c157546c,402,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4b6bcac fork_exit(c0c83630,c1a8a224,e4b6bce8) at fork_exit+0x7e/frame 0xe4b6bcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6bcd4 --- trap 0, eip = 0, esp = 0xe4b6bd20, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc8e499c0 sched_switch(c8e499c0,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b6eba0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b6ebd8 sleepq_switch(c8e499c0,0,c1583a69,279,c8f0ed00,...) at sleepq_switch+0x127/frame 0xe4b6ec0c sleepq_wait(c8f0ed00,0,c157df6a,c1566ade,0,...) at sleepq_wait+0x48/frame 0xe4b6ec30 msleep_spin_sbt(c8f0ed00,c8f0ed18,c1566ade,0,0,...) at msleep_spin_sbt+0x22d/frame 0xe4b6ec74 taskqueue_thread_loop(c1a8a224,e4b6ece8,c157546c,402,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4b6ecac fork_exit(c0c83630,c1a8a224,e4b6ece8) at fork_exit+0x7e/frame 0xe4b6ecd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b6ecd4 --- trap 0, eip = 0, esp = 0xe4b6ed20, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc8e49680 sched_switch(c8e49680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b71ba0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xe4b71bd8 sleepq_switch(c8e49680,0,c1583a69,279,c8f0ed00,...) at sleepq_switch+0x127/frame 0xe4b71c0c sleepq_wait(c8f0ed00,0,c157df6a,c1566ade,0,...) at sleepq_wait+0x48/frame 0xe4b71c30 msleep_spin_sbt(c8f0ed00,c8f0ed18,c1566ade,0,0,...) at msleep_spin_sbt+0x22d/frame 0xe4b71c74 taskqueue_thread_loop(c1a8a224,e4b71ce8,c157546c,402,0,...) at taskqueue_thread_loop+0xf3/frame 0xe4b71cac fork_exit(c0c83630,c1a8a224,e4b71ce8) at fork_exit+0x7e/frame 0xe4b71cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b71cd4 --- trap 0, eip = 0, esp = 0xe4b71d20, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc9004680 sched_switch(c9004680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b77b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b77bbc sleepq_switch(c9004680,0,c1583a69,279,c9004680,...) at sleepq_switch+0x127/frame 0xe4b77bf0 sleepq_wait(c8f0ec00,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b77c14 _sleep(c8f0ec00,c8f0ec18,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b77c74 taskqueue_thread_loop(c1a9a984,e4b77ce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b77cac fork_exit(c0c83630,c1a9a984,e4b77ce8) at fork_exit+0x7e/frame 0xe4b77cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b77cd4 --- trap 0, eip = 0, esp = 0xe4b77d20, ebp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xc9004000 sched_switch(c9004000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xe4b7db84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xe4b7dbbc sleepq_switch(c9004000,0,c1583a69,279,c9004000,...) at sleepq_switch+0x127/frame 0xe4b7dbf0 sleepq_wait(c8f0eb00,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xe4b7dc14 _sleep(c8f0eb00,c8f0eb18,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xe4b7dc74 taskqueue_thread_loop(c1a97288,e4b7dce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xe4b7dcac fork_exit(c0c83630,c1a97288,e4b7dce8) at fork_exit+0x7e/frame 0xe4b7dcd4 fork_trampoline() at fork_trampoline+0x8/frame 0xe4b7dcd4 --- trap 0, eip = 0, esp = 0xe4b7dd20, ebp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xc90e1680 sched_switch(c90e1680,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd71ba0 mi_switch(104,0,c1583a69,202,c1583a69,...) at mi_switch+0x1b4/frame 0xedd71bd8 sleepq_switch(c90e1680,0,c1583a69,279,c8f0e600,...) at sleepq_switch+0x127/frame 0xedd71c0c sleepq_wait(c8f0e600,0,c157df6a,c1566ade,0,...) at sleepq_wait+0x48/frame 0xedd71c30 msleep_spin_sbt(c8f0e600,c8f0e618,c1566ade,0,0,...) at msleep_spin_sbt+0x22d/frame 0xedd71c74 taskqueue_thread_loop(c1a88680,edd71ce8,c157546c,402,0,...) at taskqueue_thread_loop+0xf3/frame 0xedd71cac fork_exit(c0c83630,c1a88680,edd71ce8) at fork_exit+0x7e/frame 0xedd71cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd71cd4 --- trap 0, eip = 0, esp = 0xedd71d20, ebp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xc90e1000 sched_switch(c90e1000,0,104,1b4,2f9,...) at sched_switch+0x4da/frame 0xedd78b84 mi_switch(104,0,c1583a69,202,0,...) at mi_switch+0x1b4/frame 0xedd78bbc sleepq_switch(c90e1000,0,c1583a69,279,c90e1000,...) at sleepq_switch+0x127/frame 0xedd78bf0 sleepq_wait(c8f0ee80,0,c1566ade,0,0,...) at sleepq_wait+0x48/frame 0xedd78c14 _sleep(c8f0ee80,c8f0ee98,0,c1566ade,0,...) at _sleep+0x3b3/frame 0xedd78c74 taskqueue_thread_loop(c1861e24,edd78ce8,c157546c,402,0,...) at taskqueue_thread_loop+0x144/frame 0xedd78cac fork_exit(c0c83630,c1861e24,edd78ce8) at fork_exit+0x7e/frame 0xedd78cd4 fork_trampoline() at fork_trampoline+0x8/frame 0xedd78cd4 --- trap 0, eip = 0, esp = 0xedd78d20, ebp = 0 --- db> dump Physical memory: 3427 MB Dumping 184 MB: 169 153 137 121 105 89 73 57 41 25 9 Dump complete db> reset (kgdb) bt #0 doadump (textdump=0x0) at pcpu.h:206 #1 0xc055b64e in db_dump (dummy=0xc0c71b5a, dummy2=0x0, dummy3=0xffffffff, dummy4=0xf39751dc "") at ../../../ddb/db_command.c:533 #2 0xc055b43b in db_command (cmd_table=) at ../../../ddb/db_command.c:440 #3 0xc055b160 in db_command_loop () at ../../../ddb/db_command.c:493 #4 0xc055da86 in db_trap (code=) at ../../../ddb/db_main.c:251 #5 0xc0c722d7 in kdb_trap (tf=) at ../../../kern/subr_kdb.c:654 #6 0xc119731c in trap (frame=) at ../../../i386/i386/trap.c:702 #7 0xc1182c3a in calltrap () at ../../../i386/i386/exception.s:172 #8 0xc0c71b5a in kdb_enter (why=0xc157cefa "panic", msg=) at cpufunc.h:60 #9 0xc0c31bd2 in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:750 #10 0xc0c31c0b in panic (fmt=0xc1584bdc "sleeping thread") at ../../../kern/kern_shutdown.c:688 #11 0xc0c85f5b in propagate_priority (td=0xcaece340) at ../../../kern/subr_turnstile.c:226 #12 0xc0c8755b in turnstile_wait (ts=0xc96f1800, owner=0xcaece340, queue=0x0) at ../../../kern/subr_turnstile.c:742 #13 0xc0c10927 in __mtx_lock_sleep (c=0xc8e40a30, tid=0xca3d6680, opts=0x0, file=0xc1566c5f "../../../fs/pseudofs/pseudofs_internal.h", line=0x67) at ../../../kern/kern_mutex.c:533 #14 0xc0c10045 in __mtx_lock_flags (c=0xc8e40a30, opts=0x0, file=0xc1566c5f "../../../fs/pseudofs/pseudofs_internal.h", line=0x67) at ../../../kern/kern_mutex.c:226 #15 0xc0b6cba6 in pfs_lookup (va=) at pseudofs_internal.h:103 #16 0xc11c633e in VOP_CACHEDLOOKUP_APV (vop=0xc180f27c, a=0xf39758d0) at vnode_if.c:195 #17 0xc0cdab69 in vfs_cache_lookup (ap=) at vnode_if.h:80 #18 0xc11c5fae in VOP_LOOKUP_APV (vop=0xc180f27c, a=0xf3975990) at vnode_if.c:127 #19 0xc0ce463d in lookup (ndp=0xf3975ae8) at vnode_if.h:54 #20 0xc0ce3cdf in namei (ndp=) at ../../../kern/vfs_lookup.c:306 #21 0xc0cfca6e in kern_statat (td=0xca3d6680, flag=, fd=0xffffff9c, path=, sbp=0xca3d6680, hook=0xffffff9c) at ../../../kern/vfs_syscalls.c:2165 #22 0xc0cfcc39 in sys_fstatat (td=, uap=0xf3975ca8) at ../../../kern/vfs_syscalls.c:2141 #23 0xc1197f68 in syscall (frame=) at subr_syscall.c:135 #24 0xc1182cef in Xint0x80_syscall () at ../../../i386/i386/exception.s:278 #25 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 11 #11 0xc0c85f5b in propagate_priority (td=0xcaece340) at ../../../kern/subr_turnstile.c:226 226 panic("sleeping thread"); (kgdb) l 221 if (TD_IS_SLEEPING(td)) { 222 printf( 223 "Sleeping thread (tid %d, pid %d) owns a non-sleepable lock\n", 224 td->td_tid, td->td_proc->p_pid); 225 kdb_backtrace_thread(td); 226 panic("sleeping thread"); 227 } 228 229 /* 230 * If this thread already has higher priority than the (kgdb) info loc ts = (struct turnstile *) 0xc96f1800 pri = 0x91 (kgdb) p *td $1 = {td_lock = 0xc19dbae4, td_proc = 0xc9a98a08, td_plist = {tqe_next = 0x0, tqe_prev = 0xc9a98a10}, td_runq = {tqe_next = 0x0, tqe_prev = 0xc19cd544}, td_slpq = {tqe_next = 0x0, tqe_prev = 0xc899c180}, td_lockq = {tqe_next = 0xcbe71000, tqe_prev = 0xf3a8f800}, td_hash = {le_next = 0x0, le_prev = 0xc8e15e3c}, td_cpuset = 0xc8e187f8, td_sel = 0x0, td_sleepqueue = 0x0, td_turnstile = 0xc96f0600, td_rlqe = 0xc97505cc, td_umtxq = 0xc9ab1200, td_vm_dom_policy = {seq = 0x0, p = {policy = VM_POLICY_NONE, domain = 0xffffffff}}, td_tid = 0x1878f, td_sigqueue = {sq_signals = {__bits = 0xcaece398}, sq_kill = { __bits = 0xcaece3a8}, sq_list = {tqh_first = 0x0, tqh_last = 0xcaece3b8}, sq_proc = 0xc9a98a08, sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x4, td_inhibitors = 0x2, td_pflags = 0x0, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0xc19be8c1, td_wmesg = 0xc1575203 "Sched fuzz fail point", td_owepreempt = 0x0, td_tsqueue = 0xff, td_locks = 0x3, td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_stopsched = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0xc96f1800}, td_sleeplocks = 0xc1a09470, td_intr_nesting_level = 0x0, td_pinned = 0x0, td_ucred = 0xc95f3c00, td_limit = 0xc9aa7100, td_estcpu = 0x0, td_slptick = 0x7ffbb9c3, td_blktick = 0x0, td_swvoltick = 0x7ffbb9c3, td_cow = 0xf, td_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0xb38, ru_ixrss = 0x450, ru_idrss = 0x170, ru_isrss = 0x1700, ru_minflt = 0xd5, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x1, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x3ba, ru_nivcsw = 0x84}, td_rux = {rux_runtime = 0x0, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x0, rux_tu = 0x0}, td_incruntime = 0x25705192, td_runtime = 0x25705192, td_pticks = 0xa, td_sticks = 0x29, td_iticks = 0x0, td_uticks = 0x5, td_intrval = 0x0, td_oldsigmask = {__bits = 0xcaece4cc}, td_generation = 0x43e, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x4}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = 0xcaece4f8 "ls", td_fpop = 0x0, td_dbgflags = 0x0, td_dbgksi = {ksi_link = { tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0x0}, _timer = {_timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = 0xcaece540}}}, ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0, td_dbg_forked = 0x0, td_vp_reserv = 0x0, td_no_sleeping = 0x0, td_dom_rr_idx = 0x0, td_su = 0x0, td_sigmask = {__bits = 0xcaece590}, td_rqindex = 0x24, td_base_pri = 0x93, td_priority = 0x93, td_pri_class = 0x3, td_user_pri = 0xad, td_base_user_pri = 0xad, td_dbg_sc_code = 0x0, td_dbg_sc_narg = 0x0, td_pcb = 0xf3a65d40, td_state = TDS_INHIBITED, td_uretoff = {tdu_retval = 0xcaece5b8, tdu_off = 0x100000000000}, td_cowgen = 0x0, td_slpcallout = {c_links = {le = {le_next = 0x0, le_prev = 0xc8eb63ac}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0xc8eb63ac}}, c_time = 0x140eba8600d, c_precision = 0xb020c3, c_arg = 0xcaece340, c_func = 0xc0c7e890 , c_lock = 0x0, c_flags = 0x2, c_iflags = 0x110, c_cpu = 0x1}, td_frame = 0xf3a65ce8, td_kstack_obj = 0xcb7e0140, td_kstack = 0xf3a64000, td_kstack_pages = 0x2, td_critnest = 0x1, td_md = {md_spinlock_count = 0x1, md_saved_flags = 0x246, md_spurflt_addr = 0x0}, td_sched = 0xcaece64c, td_ar = 0x0, td_lprof = 0xcaece618, td_dtrace = 0xca0f1300, td_errno = 0x0, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0x0, td_rfppwait_p = 0xc9a96000, td_ma = 0x0, td_ma_cnt = 0x0, td_emuldata = 0x0, td_lastcpu = 0x1, td_oncpu = 0xffffffff} (kgdb) p *td->td_proc $2 = {p_list = {le_next = 0xcbe35a08, le_prev = 0xcbdc96b0}, p_threads = {tqh_first = 0xcaece340, tqh_last = 0xcaece348}, p_slock = {lock_object = {lo_name = 0xc157a20c "process slock", lo_flags = 0x20030000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, p_ucred = 0xc95f3c00, p_fd = 0xcbdb3000, p_fdtol = 0x0, p_stats = 0xc97c2900, p_limit = 0xc9aa7100, p_limco = {c_links = {le = {le_next = 0x0, le_prev = 0x0}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0x0, c_precision = 0x0, c_arg = 0x0, c_func = 0, c_lock = 0xc9a98ab0, c_flags = 0x0, c_iflags = 0x0, c_cpu = 0x0}, p_sigacts = 0xcbdb9000, p_flag = 0x10004002, p_flag2 = 0x0, p_state = PRS_NORMAL, p_pid = 0x706, p_hash = {le_next = 0x0, le_prev = 0xc8e08c18}, p_pglist = {le_next = 0xcaee8000, le_prev = 0xc9378a88}, p_pptr = 0xc9378a08, p_sibling = {le_next = 0x0, le_prev = 0xc9378a9c}, p_children = {lh_first = 0x0}, p_reaper = 0xc8e44a08, p_reaplist = {lh_first = 0x0}, p_reapsibling = {le_next = 0xcbe35a08, le_prev = 0xcbdc9750}, p_mtx = {lock_object = {lo_name = 0xc157a1ff "process lock", lo_flags = 0x21430000, lo_data = 0x0, lo_witness = 0xc899f340}, mtx_lock = 0x4}, p_statmtx = {lock_object = { lo_name = 0xc157a21a "pstatl", lo_flags = 0x20030000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, p_itimmtx = {lock_object = {lo_name = 0xc157a221 "pitiml", lo_flags = 0x20030000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, p_profmtx = {lock_object = {lo_name = 0xc157a228 "pprofl", lo_flags = 0x20030000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, p_ksi = 0xc921ce10, p_sigqueue = { sq_signals = {__bits = 0xc9a98b04}, sq_kill = {__bits = 0xc9a98b14}, sq_list = {tqh_first = 0x0, tqh_last = 0xc9a98b24}, sq_proc = 0xc9a98a08, sq_flags = 0x1}, p_oppid = 0x0, p_vmspace = 0xc973dc30, p_swtick = 0x7ffbb0de, p_cowgen = 0x0, p_realtimer = {it_interval = {tv_sec = 0x0, tv_usec = 0x0}, it_value = {tv_sec = 0x0, tv_usec = 0x0}}, p_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = { tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0, ru_isrss = 0x0, ru_minflt = 0x0, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x0, ru_nivcsw = 0x0}, p_rux = {rux_runtime = 0x0, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x0, rux_tu = 0x0}, p_crux = { rux_runtime = 0x0, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x0, rux_tu = 0x0}, p_profthreads = 0x0, p_exitthreads = 0x0, p_traceflag = 0x0, p_tracevp = 0x0, p_tracecred = 0x0, p_textvp = 0xc9d92ca8, p_lock = 0x0, p_sigiolst = {slh_first = 0x0}, p_sigparent = 0x14, p_sig = 0x0, p_code = 0x0, p_stops = 0x0, p_stype = 0x0, p_step = 0x0, p_pfsflags = 0x0, p_nlminfo = 0x0, p_aioinfo = 0x0, p_singlethread = 0x0, p_suspcount = 0x0, p_xthread = 0x0, p_boundary_count = 0x0, p_pendingcnt = 0x0, p_itimers = 0x0, p_procdesc = 0x0, p_treeflag = 0x0, p_pendingexits = 0x0, p_magic = 0xbeefface, p_osrel = 0x10c945, p_comm = 0xc9a98c78 "ls", p_sysent = 0xc1857130, p_args = 0xc937d380, p_cpulimit = 0x7fffffffffffffff, p_nice = 0x0, p_fibnum = 0x0, p_reapsubtree = 0x329, p_xexit = 0x0, p_xsig = 0x0, p_pgrp = 0xc977e340, p_klist = {kl_list = {slh_first = 0x0}, kl_lock = 0xc0be6820 , kl_unlock = 0xc0be6860 , kl_assert_locked = 0xc0be68a0 , kl_assert_unlocked = 0xc0be68e0 , kl_lockarg = 0xc9a98ab0}, p_numthreads = 0x1, p_md = {md_ldt = 0x0}, p_itcallout = { c_links = {le = {le_next = 0x0, le_prev = 0x0}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0x0, c_precision = 0x0, c_arg = 0x0, c_func = 0, c_lock = 0xc9a98ab0, c_flags = 0x0, c_iflags = 0x0, c_cpu = 0x0}, p_acflag = 0x0, p_peers = 0x0, p_leader = 0xc9a98a08, p_emuldata = 0x0, p_label = 0x0, p_sched = 0xc9a98d60, p_ktr = {stqh_first = 0x0, stqh_last = 0xc9a98d18}, p_mqnotifier = { lh_first = 0x0}, p_dtrace = 0xc9345e40, p_pwait = {cv_description = 0xc157ada1 "ppwait", cv_waiters = 0x0}, p_dbgwait = {cv_description = 0xc157ada8 "dbgwait", cv_waiters = 0x0}, p_prev_runtime = 0x0, p_racct = 0x0, p_throttled = 0x0, p_vm_dom_policy = {seq = 0x2, p = {policy = VM_POLICY_NONE, domain = 0xffffffff}}, p_orphan = {le_next = 0x0, le_prev = 0x0}, p_orphans = {lh_first = 0x0}} (kgdb) (kgdb) f 12 #12 0xc0c8755b in turnstile_wait (ts=0xc96f1800, owner=0xcaece340, queue=0x0) at ../../../kern/subr_turnstile.c:742 742 propagate_priority(td); (kgdb) l 737 td->td_blocked = ts; 738 td->td_lockname = lock->lo_name; 739 td->td_blktick = ticks; 740 TD_SET_LOCK(td); 741 mtx_unlock_spin(&tc->tc_lock); 742 propagate_priority(td); 743 744 if (LOCK_LOG_TEST(lock, 0)) 745 CTR4(KTR_LOCK, "%s: td %d blocked on [%p] %s", __func__, 746 td->td_tid, lock, lock->lo_name); (kgdb) info loc tc = (struct turnstile_chain *) 0xc19dd2b4 td = (struct thread *) 0xca3d6680 td1 = (struct thread *) 0x0 lock = (struct lock_object *) 0xc8e40a20 (kgdb) p *lock $4 = {lo_name = 0xc1567095 "pfs_node", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xc89a6df0} (kgdb) (kgdb) f 15 #15 0xc0b6cba6 in pfs_lookup (va=) at pseudofs_internal.h:103 103 mtx_lock(&pn->pn_mutex); (kgdb) l 98 */ 99 static inline void 100 pfs_lock(struct pfs_node *pn) 101 { 102 103 mtx_lock(&pn->pn_mutex); 104 } 105 106 static inline void 107 pfs_unlock(struct pfs_node *pn) (kgdb) f 16 #16 0xc11c633e in VOP_CACHEDLOOKUP_APV (vop=0xc180f27c, a=0xf39758d0) at vnode_if.c:195 195 rc = vop->vop_cachedlookup(a); (kgdb) l 190 ASSERT_VOP_LOCKED(a->a_dvp, "VOP_CACHEDLOOKUP"); 191 KTR_START3(KTR_VOP, "VOP", "VOP_CACHEDLOOKUP", (uintptr_t)a, 192 "dvp:0x%jX", (uintptr_t)a->a_dvp, "vpp:0x%jX", a->a_vpp, "cnp:0x%jX", a->a_cnp); 193 VFS_PROLOGUE(a->a_dvp->v_mount); 194 if (vop->vop_cachedlookup != NULL) 195 rc = vop->vop_cachedlookup(a); 196 else 197 rc = vop->vop_bypass(&a->a_gen); 198 VFS_EPILOGUE(a->a_dvp->v_mount); 199 SDT_PROBE3(vfs, vop, vop_cachedlookup, return, a->a_dvp, a, rc); (kgdb) info loc mp__ = (struct mount *) 0xc95e92ec _enable_stops = 0x0 rc = 0x0 (kgdb) p *a $3 = {a_gen = {a_desc = 0xc185c674}, a_dvp = 0xcb99c5a0, a_vpp = 0xf3975b30, a_cnp = 0xf3975b44} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_clock.c =================================================================== --- /usr/src/sys/kern/kern_clock.c (revision 296564) +++ /usr/src/sys/kern/kern_clock.c (working copy) @@ -46,6 +46,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -802,6 +803,11 @@ profclock_cnt(int cnt, int usermode, uintfptr_t pc if (td->td_proc->p_flag & P_PROFIL) addupc_intr(td, pc, cnt); } + + KFAIL_POINT_CODE(_kern_sched_fuzz, sched_fuzz_scheduler, { + sched_fuzz_switch_next[curcpu] = true; + }); + #ifdef GPROF else { /* Index: /usr/src/sys/kern/kern_condvar.c =================================================================== --- /usr/src/sys/kern/kern_condvar.c (revision 296564) +++ /usr/src/sys/kern/kern_condvar.c (working copy) @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -150,6 +151,9 @@ _cv_wait(struct cv *cvp, struct lock_object *lock) } sleepq_wait(cvp, 0); + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_cv_wait, + 0, {}); + #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) ktrcsw(0, 0, cv_wmesg(cvp)); @@ -336,6 +340,9 @@ _cv_timedwait_sbt(struct cv *cvp, struct lock_obje } rval = sleepq_timedwait(cvp, 0); + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_cv_timed_wait, + 0, {}); + #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) ktrcsw(0, 0, cv_wmesg(cvp)); @@ -406,6 +413,9 @@ _cv_timedwait_sig_sbt(struct cv *cvp, struct lock_ } rval = sleepq_timedwait_sig(cvp, 0); + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_cv_timed_wait_sig, + 0, {}); + #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) ktrcsw(0, 0, cv_wmesg(cvp)); Index: /usr/src/sys/kern/kern_fail.c =================================================================== --- /usr/src/sys/kern/kern_fail.c (revision 296564) +++ /usr/src/sys/kern/kern_fail.c (working copy) @@ -57,12 +57,18 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include #include #include +#include +#include +#include +#include +#include #include #ifdef ILOG_DEFINE_FOR_FILE @@ -72,13 +78,47 @@ ILOG_DEFINE_FOR_FILE(L_ISI_FAIL_POINT, L_ILOG, fai static MALLOC_DEFINE(M_FAIL_POINT, "Fail Points", "fail points system"); #define fp_free(ptr) free(ptr, M_FAIL_POINT) #define fp_malloc(size, flags) malloc((size), M_FAIL_POINT, (flags)) +#define fs_free(ptr) fp_free(ptr) +#define fs_malloc() fp_malloc(sizeof(struct fail_point_setting), \ + M_WAITOK | M_ZERO) -static struct mtx g_fp_mtx; -MTX_SYSINIT(g_fp_mtx, &g_fp_mtx, "fail point mtx", MTX_DEF); -#define FP_LOCK() mtx_lock(&g_fp_mtx) -#define FP_UNLOCK() mtx_unlock(&g_fp_mtx) + /** + * These define the wchans that are used for sleeping, pausing respectively. + * They are chosen arbitrarily but need to be distinct to the failpoint and + * the sleep/pause distinction. + */ +#define FP_SLEEP_CHANNEL(fp) (void*)(fp) +#define FP_PAUSE_CHANNEL(fp) __DEVOLATILE(void*, &fp->fp_setting) /** + * Don't allow more than this many entries in a fail point set by sysctl. + * The 99.99...% case is to have 1 entry. I can't imagine having this many + * entries, so it should not limit us. Saves on re-mallocs while holding + * a non-sleepable lock. + */ +#define FP_MAX_ENTRY_COUNT 20 + +/* Used to drain sbufs to the sysctl output */ +int fail_sysctl_drain_func(void *, const char *, int); + +/* Head of tailq of struct fail_point_entry */ +TAILQ_HEAD(fail_point_entry_queue, fail_point_entry); + +/** + * fp entries garbage list; outstanding entries are cleaned up in the + * garbage collector + */ +STAILQ_HEAD(fail_point_setting_garbage, fail_point_setting); +static struct fail_point_setting_garbage fp_setting_garbage = + STAILQ_HEAD_INITIALIZER(fp_setting_garbage); +static struct mtx mtx_garbage_list; +MTX_SYSINIT(mtx_garbage_list, &mtx_garbage_list, "fail point garbage mtx", + MTX_SPIN); + +static struct sx sx_fp_set; +SX_SYSINIT(sx_fp_set, &sx_fp_set, "fail point set sx"); + +/** * Failpoint types. * Don't change these without changing fail_type_strings in fail.c. * @ingroup failpoint_private @@ -90,7 +130,14 @@ enum fail_point_t { FAIL_POINT_BREAK, /**< break into the debugger */ FAIL_POINT_PRINT, /**< print a message */ FAIL_POINT_SLEEP, /**< sleep for some msecs */ - FAIL_POINT_NUMTYPES + FAIL_POINT_PAUSE, /**< sleep until failpoint is set to off */ + FAIL_POINT_YIELD, /**< yield the cpu */ + FAIL_POINT_DELAY, /**< busy wait the cpu */ + FAIL_POINT_SCHED_FUZZ, /**< similar to sleep, but designed for schedule + * fuzzer to prevent e.g. recursing sleeps. + */ + FAIL_POINT_NUMTYPES, + FAIL_POINT_INVALID = -1 }; static struct { @@ -104,53 +151,308 @@ static struct { [FAIL_POINT_BREAK] = FP_TYPE_NM_LEN("break"), [FAIL_POINT_PRINT] = FP_TYPE_NM_LEN("print"), [FAIL_POINT_SLEEP] = FP_TYPE_NM_LEN("sleep"), + [FAIL_POINT_PAUSE] = FP_TYPE_NM_LEN("pause"), + [FAIL_POINT_YIELD] = FP_TYPE_NM_LEN("yield"), + [FAIL_POINT_DELAY] = FP_TYPE_NM_LEN("delay"), + [FAIL_POINT_SCHED_FUZZ] = FP_TYPE_NM_LEN("schedfuzz"), }; +#define FE_COUNT_UNTRACKED (INT_MIN) + /** * Internal structure tracking a single term of a complete failpoint. * @ingroup failpoint_private */ struct fail_point_entry { - enum fail_point_t fe_type; /**< type of entry */ + volatile bool fe_stale; + enum fail_point_t fe_type; /**< type of entry */ int fe_arg; /**< argument to type (e.g. return value) */ int fe_prob; /**< likelihood of firing in millionths */ - int fe_count; /**< number of times to fire, 0 means always */ + int fe_count; /**< number of times to fire, -1 means infinite */ pid_t fe_pid; /**< only fail for this process */ - TAILQ_ENTRY(fail_point_entry) fe_entries; /**< next entry in fail point */ + struct fail_point *fe_parent; /**< backpointer to fp */ + TAILQ_ENTRY(fail_point_entry) fe_entries; /**< next entry ptr */ }; +struct fail_point_setting { + STAILQ_ENTRY(fail_point_setting) fs_garbage_link; + struct fail_point_entry_queue fp_entry_queue; + struct fail_point * fs_parent; + struct mtx feq_mtx; /* Gives fail_point_pause something to do. */ +}; + +/** + * Defines stating the equivalent of probablilty one (100%) + */ +enum { + PROB_MAX = 1000000, /* probability between zero and this number */ + PROB_DIGITS = 6 /* number of zero's in above number */ +}; + +/* Get a ref on an fp's fp_setting */ +static inline struct fail_point_setting *fail_point_setting_get_ref( + struct fail_point *fp); +/* Release a ref on an fp_setting */ +static inline void fail_point_setting_release_ref(struct fail_point *fp); +/* Allocate and initialize a struct fail_point_setting */ +static struct fail_point_setting *fail_point_setting_new(struct + fail_point *); +/* Free a struct fail_point_setting */ +static void fail_point_setting_destroy(struct fail_point_setting *fp_setting); +/* Allocate and initialize a struct fail_point_entry */ +static struct fail_point_entry *fail_point_entry_new(struct + fail_point_setting *); +/* Free a struct fail_point_entry */ +static void fail_point_entry_destroy(struct fail_point_entry *fp_entry); +/* Append fp setting to garbage list */ +static inline void fail_point_setting_garbage_append( + struct fail_point_setting *fp_setting); +/* Swap fp's setting with fp_setting_new */ +static inline struct fail_point_setting * + fail_point_swap_settings(struct fail_point *fp, + struct fail_point_setting *fp_setting_new); +/* Free up any zero-ref setting in the garbage queue */ +static void fail_point_garbage_collect(void); +/* If this fail point's setting are empty, then swap it out to NULL. */ +static inline void fail_point_eval_swap_out(struct fail_point *fp, + struct fail_point_setting *fp_setting); + +bool +fail_point_is_off(struct fail_point *fp) +{ + bool return_val; + struct fail_point_setting *fp_setting; + struct fail_point_entry *ent; + + return_val = true; + + fp_setting = fail_point_setting_get_ref(fp); + if (fp_setting != NULL) { + TAILQ_FOREACH(ent, &fp_setting->fp_entry_queue, + fe_entries) { + if (!ent->fe_stale) { + return_val = false; + break; + } + } + } + fail_point_setting_release_ref(fp); + + return (return_val); +} + +/* Allocate and initialize a struct fail_point_setting */ +static struct fail_point_setting * +fail_point_setting_new(struct fail_point *fp) +{ + struct fail_point_setting *fs_new; + + fs_new = fs_malloc(); + fs_new->fs_parent = fp; + TAILQ_INIT(&fs_new->fp_entry_queue); + mtx_init(&fs_new->feq_mtx, "fail point entries", NULL, MTX_SPIN); + + fail_point_setting_garbage_append(fs_new); + + return (fs_new); +} + +/* Free a struct fail_point_setting */ +static void +fail_point_setting_destroy(struct fail_point_setting *fp_setting) +{ + struct fail_point_entry *ent; + + while (!TAILQ_EMPTY(&fp_setting->fp_entry_queue)) { + ent = TAILQ_FIRST(&fp_setting->fp_entry_queue); + TAILQ_REMOVE(&fp_setting->fp_entry_queue, ent, fe_entries); + fail_point_entry_destroy(ent); + } + + fs_free(fp_setting); +} + +/* Allocate and initialize a struct fail_point_entry */ +static struct fail_point_entry * +fail_point_entry_new(struct fail_point_setting *fp_setting) +{ + struct fail_point_entry *fp_entry; + + fp_entry = fp_malloc(sizeof(struct fail_point_entry), + M_WAITOK | M_ZERO); + fp_entry->fe_parent = fp_setting->fs_parent; + fp_entry->fe_prob = PROB_MAX; + fp_entry->fe_pid = NO_PID; + fp_entry->fe_count = FE_COUNT_UNTRACKED; + TAILQ_INSERT_TAIL(&fp_setting->fp_entry_queue, fp_entry, + fe_entries); + + return (fp_entry); +} + +/* Free a struct fail_point_entry */ +static void +fail_point_entry_destroy(struct fail_point_entry *fp_entry) +{ + + fp_free(fp_entry); +} + +/* Get a ref on an fp's fp_setting */ +static inline struct fail_point_setting * +fail_point_setting_get_ref(struct fail_point *fp) +{ + struct fail_point_setting *fp_setting; + + /* Invariant: if we have a ref, our pointer to fp_setting is safe */ + atomic_add_acq_32(&fp->fp_ref_cnt, 1); + fp_setting = fp->fp_setting; + + return (fp_setting); +} + +/* Release a ref on an fp_setting */ static inline void -fail_point_sleep(struct fail_point *fp, struct fail_point_entry *ent, - int msecs, enum fail_point_return_code *pret) +fail_point_setting_release_ref(struct fail_point *fp) { - /* convert from millisecs to ticks, rounding up */ - int timo = ((msecs * hz) + 999) / 1000; + KASSERT(&fp->fp_ref_cnt > 0, ("Attempting to deref w/no refs")); + atomic_subtract_rel_32(&fp->fp_ref_cnt, 1); +} + +/* Append fp entries to fp garbage list */ +static inline void +fail_point_setting_garbage_append(struct fail_point_setting *fp_setting) +{ + + mtx_lock_spin(&mtx_garbage_list); + STAILQ_INSERT_TAIL(&fp_setting_garbage, fp_setting, + fs_garbage_link); + mtx_unlock_spin(&mtx_garbage_list); +} + +/* Swap fp's entries with fp_setting_new */ +static struct fail_point_setting * +fail_point_swap_settings(struct fail_point *fp, + struct fail_point_setting *fp_setting_new) +{ + struct fail_point_setting *fp_setting_old; + + fp_setting_old = fp->fp_setting; + fp->fp_setting = fp_setting_new; + + return (fp_setting_old); +} + +static inline void +fail_point_eval_swap_out(struct fail_point *fp, + struct fail_point_setting *fp_setting) +{ + + /* We may have already been swapped out and replaced; ignore. */ + if (fp->fp_setting == fp_setting) + fail_point_swap_settings(fp, NULL); +} + +/* Free up any zero-ref entries in the garbage queue */ +static void +fail_point_garbage_collect() +{ + struct fail_point_setting *fs_current, *fs_next; + struct fail_point_setting_garbage fp_ents_free_list; + + /** + * We will transfer the entries to free to fp_ents_free_list while holding + * the spin mutex, then free it after we drop the lock. This avoids + * triggering witness due to sleepable mutexes in the memory + * allocator. + */ + STAILQ_INIT(&fp_ents_free_list); + + mtx_lock_spin(&mtx_garbage_list); + STAILQ_FOREACH_SAFE(fs_current, &fp_setting_garbage, fs_garbage_link, + fs_next) { + if (fs_current->fs_parent->fp_setting != fs_current && + fs_current->fs_parent->fp_ref_cnt == 0) { + STAILQ_REMOVE(&fp_setting_garbage, fs_current, + fail_point_setting, fs_garbage_link); + STAILQ_INSERT_HEAD(&fp_ents_free_list, fs_current, + fs_garbage_link); + } + } + mtx_unlock_spin(&mtx_garbage_list); + + STAILQ_FOREACH_SAFE(fs_current, &fp_ents_free_list, fs_garbage_link, + fs_next) + fail_point_setting_destroy(fs_current); +} + +/* Drain out all refs from this fail point */ +static inline void +fail_point_drain(struct fail_point *fp, int expected_ref) +{ + struct fail_point_setting *entries; + + entries = fail_point_swap_settings(fp, NULL); + /** + * We have unpaused all threads; so we will wait no longer + * than the time taken for the longest remaining sleep, or + * the length of time of a long-running code block. + */ + while (fp->fp_ref_cnt > expected_ref) { + wakeup(FP_PAUSE_CHANNEL(fp)); + tsleep(&fp, PWAIT, "fail_point_drain", hz / 100); + } + fail_point_swap_settings(fp, entries); +} + +static inline void +fail_point_pause(struct fail_point *fp, enum fail_point_return_code *pret, + struct mtx *mtx_sleep) +{ + + if (fp->fp_pre_sleep_fn) + fp->fp_pre_sleep_fn(fp->fp_pre_sleep_arg); + + msleep_spin(FP_PAUSE_CHANNEL(fp), mtx_sleep, "failpt", 0); + + if (fp->fp_post_sleep_fn) + fp->fp_post_sleep_fn(fp->fp_post_sleep_arg); +} + +static inline void +fail_point_sleep(struct fail_point *fp, int msecs, + enum fail_point_return_code *pret) +{ + int timo; + + /* Convert from millisecs to ticks, rounding up */ + timo = howmany(msecs * hz, 1000); + if (timo > 0) { - if (fp->fp_sleep_fn == NULL) { - msleep(fp, &g_fp_mtx, PWAIT, "failpt", timo); + if (!(fp->fp_flags & FAIL_POINT_USE_TIMEOUT_PATH)) { + if (fp->fp_pre_sleep_fn) + fp->fp_pre_sleep_fn(fp->fp_pre_sleep_arg); + + tsleep(FP_SLEEP_CHANNEL(fp), PWAIT, "failpt", timo); + + if (fp->fp_post_sleep_fn) + fp->fp_post_sleep_fn(fp->fp_post_sleep_arg); } else { - timeout(fp->fp_sleep_fn, fp->fp_sleep_arg, timo); + if (fp->fp_pre_sleep_fn) + fp->fp_pre_sleep_fn(fp->fp_pre_sleep_arg); + + timeout(fp->fp_post_sleep_fn, fp->fp_post_sleep_arg, + timo); *pret = FAIL_POINT_RC_QUEUED; } } } - -/** - * Defines stating the equivalent of probablilty one (100%) - */ -enum { - PROB_MAX = 1000000, /* probability between zero and this number */ - PROB_DIGITS = 6, /* number of zero's in above number */ -}; - -static char *parse_fail_point(struct fail_point_entries *, char *); -static char *parse_term(struct fail_point_entries *, char *); +static char *parse_fail_point(struct fail_point_setting *, char *); +static char *parse_term(struct fail_point_setting *, char *); static char *parse_number(int *out_units, int *out_decimal, char *); static char *parse_type(struct fail_point_entry *, char *); -static void free_entry(struct fail_point_entries *, struct fail_point_entry *); -static void clear_entries(struct fail_point_entries *); /** * Initialize a fail_point. The name is formed in a printf-like fashion @@ -167,7 +469,7 @@ fail_point_init(struct fail_point *fp, const char char *name; int n; - TAILQ_INIT(&fp->fp_entries); + fp->fp_setting = NULL; fp->fp_flags = 0; /* Figure out the size of the name. */ @@ -185,13 +487,16 @@ fail_point_init(struct fail_point *fp, const char fp->fp_name = name; fp->fp_location = ""; fp->fp_flags |= FAIL_POINT_DYNAMIC_NAME; - fp->fp_sleep_fn = NULL; - fp->fp_sleep_arg = NULL; + fp->fp_pre_sleep_fn = NULL; + fp->fp_pre_sleep_arg = NULL; + fp->fp_post_sleep_fn = NULL; + fp->fp_post_sleep_arg = NULL; } /** - * Free the resources held by a fail_point. - * + * Free the resources held by a fail_point, and wake any paused threads. + * Thou shalt not allow threads to hit this fail point after you enter this + * function, nor shall you call this multiple times for a given fp. * @ingroup failpoint */ void @@ -198,12 +503,17 @@ void fail_point_destroy(struct fail_point *fp) { + fail_point_drain(fp, 0); + if ((fp->fp_flags & FAIL_POINT_DYNAMIC_NAME) != 0) { fp_free(__DECONST(void *, fp->fp_name)); fp->fp_name = NULL; } fp->fp_flags = 0; - clear_entries(&fp->fp_entries); + + sx_xlock(&sx_fp_set); + fail_point_garbage_collect(); + sx_xunlock(&sx_fp_set); } /** @@ -216,21 +526,51 @@ fail_point_destroy(struct fail_point *fp) enum fail_point_return_code fail_point_eval_nontrivial(struct fail_point *fp, int *return_value) { - enum fail_point_return_code ret = FAIL_POINT_RC_CONTINUE; - struct fail_point_entry *ent, *next; + bool execute = false; + struct fail_point_entry *ent; + struct fail_point_setting *fp_setting; + enum fail_point_return_code ret; + int cont; + int count; int msecs; + int usecs; - FP_LOCK(); + ret = FAIL_POINT_RC_CONTINUE; + cont = 0; /* don't continue by default */ - TAILQ_FOREACH_SAFE(ent, &fp->fp_entries, fe_entries, next) { - int cont = 0; /* don't continue by default */ + fp_setting = fail_point_setting_get_ref(fp); + if (fp_setting == NULL) + goto abort; + TAILQ_FOREACH(ent, &fp_setting->fp_entry_queue, fe_entries) { + + if (ent->fe_stale) + continue; + if (ent->fe_prob < PROB_MAX && ent->fe_prob < random() % PROB_MAX) continue; + if (ent->fe_pid != NO_PID && ent->fe_pid != curproc->p_pid) continue; + if (ent->fe_count != FE_COUNT_UNTRACKED) { + count = ent->fe_count; + while (count > 0) { + if (atomic_cmpset_32(&ent->fe_count, count, count - 1)) { + count--; + execute = true; + break; + } + count = ent->fe_count; + } + if (execute == false) + /* We lost the race; consider the entry stale and bail now */ + continue; + if (count == 0) + ent->fe_stale = true; + } + switch (ent->fe_type) { case FAIL_POINT_PANIC: panic("fail point %s panicking", fp->fp_name); @@ -244,7 +584,7 @@ fail_point_eval_nontrivial(struct fail_point *fp, case FAIL_POINT_BREAK: printf("fail point %s breaking to debugger\n", - fp->fp_name); + fp->fp_name); breakpoint(); break; @@ -254,38 +594,70 @@ fail_point_eval_nontrivial(struct fail_point *fp, break; case FAIL_POINT_SLEEP: - /* - * Free the entry now if necessary, since - * we're about to drop the mutex and sleep. - */ msecs = ent->fe_arg; - if (ent->fe_count > 0 && --ent->fe_count == 0) { - free_entry(&fp->fp_entries, ent); - ent = NULL; - } - if (msecs) - fail_point_sleep(fp, ent, msecs, &ret); + fail_point_sleep(fp, msecs, &ret); break; + case FAIL_POINT_PAUSE: + /** + * Pausing is inherently strange with multiple + * entries given our design. That is because some + * entries could be unreachable, for instance in cases like: + * pause->return. We can never reach the return entry. + * The sysctl layer actually truncates all entries after + * a pause for this reason. + */ + mtx_lock_spin(&fp_setting->feq_mtx); + fail_point_pause(fp, &ret, &fp_setting->feq_mtx); + mtx_unlock_spin(&fp_setting->feq_mtx); + break; + + case FAIL_POINT_YIELD: + kern_yield(-1); + break; + + case FAIL_POINT_DELAY: + usecs = ent->fe_arg; + DELAY(usecs); + break; + + case FAIL_POINT_SCHED_FUZZ: + if (fp->fp_flags & FAIL_POINT_NONSLEEPABLE) { + if (ent->fe_arg <= 1) + usecs = 1; + else + usecs = random() % ( + ent->fe_arg - 1) + 1; + DELAY(usecs); + } else { + if (ent->fe_arg <= 1) + msecs = 1; + else + msecs = random() % ( + ent->fe_arg - 1) + 1; + pause_sched_fuzz( + "Sched fuzz fail point", + msecs); + } + break; + default: break; } - if (ent != NULL && ent->fe_count > 0 && --ent->fe_count == 0) - free_entry(&fp->fp_entries, ent); if (cont == 0) break; } - /* Get rid of "off"s at the end. */ - while ((ent = TAILQ_LAST(&fp->fp_entries, fail_point_entries)) && - ent->fe_type == FAIL_POINT_OFF) - free_entry(&fp->fp_entries, ent); + if (fail_point_is_off(fp)) + fail_point_eval_swap_out(fp, fp_setting); - FP_UNLOCK(); +abort: + fail_point_setting_release_ref(fp); return (ret); + } /** @@ -292,13 +664,45 @@ fail_point_eval_nontrivial(struct fail_point *fp, * Translate internal fail_point structure into human-readable text. */ static void -fail_point_get(struct fail_point *fp, struct sbuf *sb) +fail_point_get(struct fail_point *fp, struct sbuf *sb, + bool verbose) { struct fail_point_entry *ent; + struct fail_point_setting *fp_setting; + struct fail_point_entry *fp_entry_cpy; + int cnt_sleeping; + int idx; + int printed_entry_count; - FP_LOCK(); + cnt_sleeping = 0; + idx = 0; + printed_entry_count = 0; - TAILQ_FOREACH(ent, &fp->fp_entries, fe_entries) { + fp_entry_cpy = fp_malloc(sizeof(struct fail_point_entry) * + (FP_MAX_ENTRY_COUNT + 1), M_WAITOK); + + fp_setting = fail_point_setting_get_ref(fp); + + if (fp_setting != NULL) { + TAILQ_FOREACH(ent, &fp_setting->fp_entry_queue, fe_entries) { + if (ent->fe_stale) + continue; + + KASSERT(printed_entry_count < FP_MAX_ENTRY_COUNT, + ("FP entry list larger than allowed")); + + fp_entry_cpy[printed_entry_count] = *ent; + ++printed_entry_count; + } + } + fail_point_setting_release_ref(fp); + + /* This is our equivalent of a NULL terminator */ + fp_entry_cpy[printed_entry_count].fe_type = FAIL_POINT_INVALID; + + while (idx < printed_entry_count) { + ent = &fp_entry_cpy[idx]; + ++idx; if (ent->fe_prob < PROB_MAX) { int decimal = ent->fe_prob % (PROB_MAX / 100); int units = ent->fe_prob / (PROB_MAX / 100); @@ -313,7 +717,7 @@ static void } sbuf_printf(sb, "%%"); } - if (ent->fe_count > 0) + if (ent->fe_count >= 0) sbuf_printf(sb, "%d*", ent->fe_count); sbuf_printf(sb, "%s", fail_type_strings[ent->fe_type].name); if (ent->fe_arg) @@ -323,10 +727,29 @@ static void if (TAILQ_NEXT(ent, fe_entries)) sbuf_printf(sb, "->"); } - if (TAILQ_EMPTY(&fp->fp_entries)) + if (!printed_entry_count) sbuf_printf(sb, "off"); - FP_UNLOCK(); + fp_free(fp_entry_cpy); + if (verbose) { + /* Print number of sleeping threads. queue=0 is the argument + * used by msleep when sending our threads to sleep. */ + sbuf_printf(sb, "\nsleeping_thread_stacks = {\n"); + sleepq_sbuf_print_stacks(sb, FP_SLEEP_CHANNEL(fp), 0, + &cnt_sleeping); + + sbuf_printf(sb, "},\n"); + sbuf_printf(sb, "sleeping_thread_count = %d,\n", + cnt_sleeping); + + sbuf_printf(sb, "paused_thread_stacks = {\n"); + sleepq_sbuf_print_stacks(sb, FP_PAUSE_CHANNEL(fp), 0, + &cnt_sleeping); + + sbuf_printf(sb, "},\n"); + sbuf_printf(sb, "paused_thread_count = %d\n", + cnt_sleeping); + } } /** @@ -336,38 +759,91 @@ static void static int fail_point_set(struct fail_point *fp, char *buf) { - int error = 0; struct fail_point_entry *ent, *ent_next; - struct fail_point_entries new_entries; + struct fail_point_setting *entries; + bool should_wake_paused; + bool should_truncate; + int error; + error = 0; + should_wake_paused = false; + should_truncate = false; + /* Parse new entries. */ - TAILQ_INIT(&new_entries); - if (!parse_fail_point(&new_entries, buf)) { - clear_entries(&new_entries); + /** + * ref protects our new malloc'd stuff from being garbage collected + * before we link it. + */ + fail_point_setting_get_ref(fp); + entries = fail_point_setting_new(fp); + if (parse_fail_point(entries, buf) == NULL) { + STAILQ_REMOVE(&fp_setting_garbage, entries, + fail_point_setting, fs_garbage_link); + fail_point_setting_destroy(entries); error = EINVAL; goto end; } - FP_LOCK(); + /** + * Transfer the entries we are going to keep to a new list. + * Get rid of useless zero probability entries, and entries with hit + * count 0. + * If 'off' is present, and it has no hit count set, then all entries + * after it are discarded since they are unreachable. + */ + TAILQ_FOREACH_SAFE(ent, &entries->fp_entry_queue, fe_entries, ent_next) { + if (ent->fe_prob == 0 || ent->fe_count == 0) { + printf("Discarding entry which cannot execute %s\n", + fail_type_strings[ent->fe_type].name); + TAILQ_REMOVE(&entries->fp_entry_queue, ent, + fe_entries); + fp_free(ent); + continue; + } else if (should_truncate) { + printf("Discarding unreachable entry %s\n", + fail_type_strings[ent->fe_type].name); + TAILQ_REMOVE(&entries->fp_entry_queue, ent, + fe_entries); + fp_free(ent); + continue; + } - /* Move new entries in. */ - TAILQ_SWAP(&fp->fp_entries, &new_entries, fail_point_entry, fe_entries); - clear_entries(&new_entries); + if (ent->fe_type == FAIL_POINT_OFF) { + should_wake_paused = true; + if (ent->fe_count == FE_COUNT_UNTRACKED) { + should_truncate = true; + TAILQ_REMOVE(&entries->fp_entry_queue, ent, + fe_entries); + fp_free(ent); + } + } else if (ent->fe_type == FAIL_POINT_PAUSE) { + should_truncate = true; + } else if (ent->fe_type == FAIL_POINT_SLEEP && (fp->fp_flags & + FAIL_POINT_NONSLEEPABLE)) { + /** + * If this fail point is annotated as being in a + * non-sleepable ctx, convert sleep to delay and + * convert the msec argument to usecs. + */ + printf("Sleep call request on fail point in " + "non-sleepable context; using delay instead " + "of sleep\n"); + ent->fe_type = FAIL_POINT_DELAY; + ent->fe_arg *= 1000; + } + } - /* Get rid of useless zero probability entries. */ - TAILQ_FOREACH_SAFE(ent, &fp->fp_entries, fe_entries, ent_next) { - if (ent->fe_prob == 0) - free_entry(&fp->fp_entries, ent); + if (TAILQ_EMPTY(&entries->fp_entry_queue)) { + entries = fail_point_swap_settings(fp, NULL); + if (entries != NULL) + wakeup(FP_PAUSE_CHANNEL(fp)); + } else { + if (should_wake_paused) + wakeup(FP_PAUSE_CHANNEL(fp)); + fail_point_swap_settings(fp, entries); } - /* Get rid of "off"s at the end. */ - while ((ent = TAILQ_LAST(&fp->fp_entries, fail_point_entries)) && - ent->fe_type == FAIL_POINT_OFF) - free_entry(&fp->fp_entries, ent); - - FP_UNLOCK(); - - end: +end: #ifdef IWARNING if (error) IWARNING("Failed to set %s %s to %s", @@ -377,6 +853,7 @@ fail_point_set(struct fail_point *fp, char *buf) fp->fp_name, fp->fp_location, buf); #endif /* IWARNING */ + fail_point_setting_release_ref(fp); return (error); } @@ -385,25 +862,33 @@ fail_point_set(struct fail_point *fp, char *buf) /** * Handle kernel failpoint set/get. */ + int fail_point_sysctl(SYSCTL_HANDLER_ARGS) { - struct fail_point *fp = arg1; - char *buf = NULL; + struct fail_point *fp; + char *buf; + struct sbuf *sb_check; struct sbuf sb; int error; - /* Retrieving */ - sbuf_new(&sb, NULL, 128, SBUF_AUTOEXTEND | SBUF_INCLUDENUL); - fail_point_get(fp, &sb); - sbuf_trim(&sb); - error = sbuf_finish(&sb); - if (error == 0) - error = SYSCTL_OUT(req, sbuf_data(&sb), sbuf_len(&sb)); - sbuf_delete(&sb); + error = 0; + fp = arg1; + buf = NULL; + sb_check = sbuf_new(&sb, NULL, 1024, SBUF_AUTOEXTEND); + if (sb_check != &sb) + return (ENOMEM); + + sbuf_set_drain(&sb, (sbuf_drain_func *)fail_sysctl_drain_func, req); + /* Setting */ - if (!error && req->newptr) { + /** + * Lock protects any new entries from being garbage collected before we + * can link them to the fail point. + */ + sx_xlock(&sx_fp_set); + if (req->newptr) { if (req->newlen > MAX_FAIL_POINT_BUF) { error = EINVAL; goto out; @@ -417,32 +902,96 @@ fail_point_sysctl(SYSCTL_HANDLER_ARGS) buf[req->newlen] = '\0'; error = fail_point_set(fp, buf); - } + } + fail_point_garbage_collect(); + sx_xunlock(&sx_fp_set); + + /* Retrieving. */ + fail_point_get(fp, &sb, false); + out: - fp_free(buf); + sbuf_finish(&sb); + sbuf_delete(&sb); + + if (buf) + fp_free(buf); + return (error); } +int +fail_point_sysctl_status(SYSCTL_HANDLER_ARGS) +{ + struct fail_point *fp; + struct sbuf sb, *sb_check; + + fp = arg1; + + sb_check = sbuf_new(&sb, NULL, 1024, SBUF_AUTOEXTEND); + if (sb_check != &sb) + return (ENOMEM); + + sbuf_set_drain(&sb, (sbuf_drain_func *)fail_sysctl_drain_func, req); + + /* Retrieving. */ + fail_point_get(fp, &sb, true); + + sbuf_finish(&sb); + sbuf_delete(&sb); + + /** + * Lock protects any new entries from being garbage collected before we + * can link them to the fail point. + */ + sx_xlock(&sx_fp_set); + fail_point_garbage_collect(); + sx_xunlock(&sx_fp_set); + + return (0); +} + +int +fail_sysctl_drain_func(void *sysctl_args, const char *buf, int len) +{ + struct sysctl_req *sa; + int error; + + sa = sysctl_args; + + error = SYSCTL_OUT(sa, buf, len); + + if (error == ENOMEM) + return (-1); + else + return (len); +} + + /** * Internal helper function to translate a human-readable failpoint string * into a internally-parsable fail_point structure. */ static char * -parse_fail_point(struct fail_point_entries *ents, char *p) +parse_fail_point(struct fail_point_setting *ents, char *p) { /* :: * ( "->" )* */ + uint8_t term_count; + + term_count = 1; + p = parse_term(ents, p); if (p == NULL) return (NULL); + while (*p != '\0') { - if (p[0] != '-' || p[1] != '>') + term_count++; + if (p[0] != '-' || p[1] != '>' || + (p = parse_term(ents, p+2)) == NULL || + term_count > FP_MAX_ENTRY_COUNT) return (NULL); - p = parse_term(ents, p + 2); - if (p == NULL) - return (NULL); } return (p); } @@ -451,14 +1000,11 @@ static char * * Internal helper function to parse an individual term from a failpoint. */ static char * -parse_term(struct fail_point_entries *ents, char *p) +parse_term(struct fail_point_setting *ents, char *p) { struct fail_point_entry *ent; - ent = fp_malloc(sizeof *ent, M_WAITOK | M_ZERO); - ent->fe_prob = PROB_MAX; - ent->fe_pid = NO_PID; - TAILQ_INSERT_TAIL(ents, ent, fe_entries); + ent = fail_point_entry_new(ents); /* * :: @@ -483,7 +1029,7 @@ static char * if (ent->fe_prob > PROB_MAX) ent->fe_prob = PROB_MAX; } else if (*p == '*') { - if (!units || decimal) + if (!units || units < 0 || decimal) return (NULL); ent->fe_count = units; } else @@ -500,7 +1046,7 @@ static char * /* [ "(" ")" ] */ if (*p != '(') - return p; + return (p); p++; if (!isdigit(*p) && *p != '-') return (NULL); @@ -509,7 +1055,7 @@ static char * return (NULL); /* [ "[pid " "]" ] */ -#define PID_STRING "[pid " +#define PID_STRING "[pid " if (strncmp(p, PID_STRING, sizeof(PID_STRING) - 1) != 0) return (p); p += sizeof(PID_STRING) - 1; @@ -530,7 +1076,7 @@ parse_number(int *out_units, int *out_decimal, cha { char *old_p; - /* + /** * :: * [ "." ] | * "." @@ -584,29 +1130,17 @@ parse_type(struct fail_point_entry *ent, char *beg return (NULL); } -/** - * Internal helper function to free an individual failpoint term. - */ -static void -free_entry(struct fail_point_entries *ents, struct fail_point_entry *ent) -{ - TAILQ_REMOVE(ents, ent, fe_entries); - fp_free(ent); -} +/* The fail point sysctl tree. */ +SYSCTL_NODE(_debug, OID_AUTO, fail_point, CTLFLAG_RW, 0, "fail points"); -/** - * Internal helper function to clear out all failpoint terms for a single - * failpoint. - */ -static void -clear_entries(struct fail_point_entries *ents) +/* Debugging/testing stuff for fail point */ +static int +sysctl_test_fail_point(SYSCTL_HANDLER_ARGS) { - struct fail_point_entry *ent, *ent_next; - TAILQ_FOREACH_SAFE(ent, ents, fe_entries, ent_next) - fp_free(ent); - TAILQ_INIT(ents); + KFAIL_POINT_RETURN(DEBUG_FP, test_fail_point); + return (0); } - -/* The fail point sysctl tree. */ -SYSCTL_NODE(_debug, OID_AUTO, fail_point, CTLFLAG_RW, 0, "fail points"); +SYSCTL_OID(_debug_fail_point, OID_AUTO, test_trigger_fail_point, + CTLTYPE_STRING | CTLFLAG_RD, NULL, 0, sysctl_test_fail_point, "A", + "Trigger test fail points"); Index: /usr/src/sys/kern/kern_mutex.c =================================================================== --- /usr/src/sys/kern/kern_mutex.c (revision 296564) +++ /usr/src/sys/kern/kern_mutex.c (working copy) @@ -45,6 +45,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -219,6 +220,9 @@ __mtx_lock_flags(volatile uintptr_t *c, int opts, WITNESS_CHECKORDER(&m->lock_object, (opts & ~MTX_RECURSE) | LOP_NEWORDER | LOP_EXCLUSIVE, file, line, NULL); + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_mtx_lock, + FAIL_POINT_NONSLEEPABLE, {}); + __mtx_lock(m, curthread, opts, file, line); LOCK_LOG_LOCK("LOCK", &m->lock_object, opts, m->mtx_recurse, file, line); @@ -248,6 +252,10 @@ __mtx_unlock_flags(volatile uintptr_t *c, int opts mtx_assert(m, MA_OWNED); __mtx_unlock(m, curthread, opts, file, line); + + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_mtx_unlock, + FAIL_POINT_NONSLEEPABLE, {}); + TD_LOCKS_DEC(curthread); } @@ -275,6 +283,10 @@ __mtx_lock_spin_flags(volatile uintptr_t *c, int o opts &= ~MTX_RECURSE; WITNESS_CHECKORDER(&m->lock_object, opts | LOP_NEWORDER | LOP_EXCLUSIVE, file, line, NULL); + + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_mtx_spin_lock, + FAIL_POINT_NONSLEEPABLE, {}); + __mtx_lock_spin(m, curthread, opts, file, line); LOCK_LOG_LOCK("LOCK", &m->lock_object, opts, m->mtx_recurse, file, line); @@ -303,6 +315,9 @@ __mtx_unlock_spin_flags(volatile uintptr_t *c, int mtx_assert(m, MA_OWNED); __mtx_unlock_spin(m); + + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_mtx_spin_unlock, + FAIL_POINT_NONSLEEPABLE, {}); } /* Index: /usr/src/sys/kern/kern_sx.c =================================================================== --- /usr/src/sys/kern/kern_sx.c (revision 296564) +++ /usr/src/sys/kern/kern_sx.c (working copy) @@ -45,6 +45,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include #include @@ -257,6 +258,13 @@ _sx_slock(struct sx *sx, int opts, const char *fil KASSERT(sx->sx_lock != SX_LOCK_DESTROYED, ("sx_slock() of destroyed sx @ %s:%d", file, line)); WITNESS_CHECKORDER(&sx->lock_object, LOP_NEWORDER, file, line, NULL); + + KFAIL_POINT_CODE_COND(_kern_sched_fuzz, sched_fuzz_sx_slock_delay, + (sx->lock_object.lo_flags & SX_NOADAPTIVE), FAIL_POINT_NONSLEEPABLE, + {}); + KFAIL_POINT_CODE_COND(_kern_sched_fuzz, sched_fuzz_sx_slock_sleep, + !(sx->lock_object.lo_flags & SX_NOADAPTIVE), 0, {}); + error = __sx_slock(sx, opts, file, line); if (!error) { LOCK_LOG_LOCK("SLOCK", &sx->lock_object, 0, 0, file, line); @@ -313,6 +321,13 @@ _sx_xlock(struct sx *sx, int opts, const char *fil ("sx_xlock() of destroyed sx @ %s:%d", file, line)); WITNESS_CHECKORDER(&sx->lock_object, LOP_NEWORDER | LOP_EXCLUSIVE, file, line, NULL); + + KFAIL_POINT_CODE_COND(_kern_sched_fuzz, sched_fuzz_sx_xlock_delay, + (sx->lock_object.lo_flags & SX_NOADAPTIVE), FAIL_POINT_NONSLEEPABLE, + {}); + KFAIL_POINT_CODE_COND(_kern_sched_fuzz, sched_fuzz_sx_xlock_sleep, + !(sx->lock_object.lo_flags & SX_NOADAPTIVE), 0, {}); + error = __sx_xlock(sx, curthread, opts, file, line); if (!error) { LOCK_LOG_LOCK("XLOCK", &sx->lock_object, 0, sx->sx_recurse, Index: /usr/src/sys/kern/kern_synch.c =================================================================== --- /usr/src/sys/kern/kern_synch.c (revision 296564) +++ /usr/src/sys/kern/kern_synch.c (working copy) @@ -43,6 +43,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -237,6 +238,10 @@ _sleep(void *ident, struct lock_object *lock, int sleepq_wait(ident, pri); rval = 0; } + + KFAIL_POINT_CODE_COND(_kern_sched_fuzz, sched_fuzz_sleepq, + !(priority & PSCHEDFUZZING), 0, {}); + #ifdef KTRACE if (KTRPOINT(td, KTR_CSW)) ktrcsw(0, 0, wmesg); @@ -336,7 +341,8 @@ msleep_spin_sbt(void *ident, struct mtx *mtx, cons * to a "timo" value of one. */ int -pause_sbt(const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags) +pause_sbt_pri(const char *wmesg, int priority, sbintime_t sbt, + sbintime_t pr, int flags) { KASSERT(sbt >= 0, ("pause: timeout must be >= 0")); @@ -359,9 +365,16 @@ int DELAY(sbt); return (0); } - return (_sleep(&pause_wchan[curcpu], NULL, 0, wmesg, sbt, pr, flags)); + return (_sleep(&pause_wchan[curcpu], NULL, priority, wmesg, sbt, pr, + flags)); } +int +pause_sbt(const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags) +{ + return (pause_sbt_pri(wmesg, 0, sbt, pr, flags)); +} + /* * Make all threads sleeping on the specified identifier runnable. */ Index: /usr/src/sys/kern/sched_ule.c =================================================================== --- /usr/src/sys/kern/sched_ule.c (revision 296564) +++ /usr/src/sys/kern/sched_ule.c (working copy) @@ -271,7 +271,18 @@ static int affinity; static int steal_idle = 1; static int steal_thresh = 2; +static int sched_fuzz_on = 0; + /* + * When a CPU's bit is 1, then * the next thread running on that CPU + * should switch. Indexed by cpu id. + * + * Set in kern_clock.c + */ +bool sched_fuzz_switch_next[MAXCPU] = {0}; + + +/* * One thread queue per processor. */ static struct tdq tdq_cpu[MAXCPU]; @@ -2282,7 +2293,9 @@ sched_clock(struct thread *td) * Force a context switch if the current thread has used up a full * time slice (default is 100ms). */ - if (!TD_IS_IDLETHREAD(td) && ++ts->ts_slice >= tdq_slice(tdq)) { + if ((!TD_IS_IDLETHREAD(td) && ++ts->ts_slice >= tdq_slice(tdq)) || + sched_fuzz_switch_next[curcpu]) { + sched_fuzz_switch_next[curcpu] = false; ts->ts_slice = 0; td->td_flags |= TDF_NEEDRESCHED | TDF_SLICEEND; } @@ -2865,6 +2878,27 @@ sysctl_kern_sched_topology_spec(SYSCTL_HANDLER_ARG #endif static int +sysctl_kern_sched_fuzz_prof_on(SYSCTL_HANDLER_ARGS) +{ + int error, new_val = sched_fuzz_on; + error = SYSCTL_IN(req, &new_val, sizeof(new_val)); + if (error != 0) + return (error); + + if (!sched_fuzz_on && new_val) { + sched_fuzz_on = 1; + cpu_startprofclock(); + } else if (sched_fuzz_on && !new_val) { + sched_fuzz_on = 0; + cpu_stopprofclock(); + } + + SYSCTL_OUT(req, &sched_fuzz_on, sizeof(sched_fuzz_on)); + + return (0); +} + +static int sysctl_kern_quantum(SYSCTL_HANDLER_ARGS) { int error, new_val, period; @@ -2884,6 +2918,12 @@ sysctl_kern_quantum(SYSCTL_HANDLER_ARGS) } SYSCTL_NODE(_kern, OID_AUTO, sched, CTLFLAG_RW, 0, "Scheduler"); +SYSCTL_NODE(_kern, OID_AUTO, sched_fuzz, CTLFLAG_RW, 0, + "Fuzz sched timings"); +SYSCTL_PROC(_kern_sched_fuzz, OID_AUTO, prof_on, CTLTYPE_INT | CTLFLAG_RW, + NULL, 0, sysctl_kern_sched_fuzz_prof_on, "I", + "Set to 1 to turn on sched fuzz profiling clock, 0 to turn it off; " + "this is needed for sched_fuzz_scheduler to work."); SYSCTL_STRING(_kern_sched, OID_AUTO, name, CTLFLAG_RD, "ULE", 0, "Scheduler name"); SYSCTL_PROC(_kern_sched, OID_AUTO, quantum, CTLTYPE_INT | CTLFLAG_RW, Index: /usr/src/sys/kern/subr_sleepqueue.c =================================================================== --- /usr/src/sys/kern/subr_sleepqueue.c (revision 296564) +++ /usr/src/sys/kern/subr_sleepqueue.c (working copy) @@ -75,6 +75,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -83,6 +84,7 @@ __FBSDID("$FreeBSD$"); #include #endif + /* * Constants for the hash table of sleep queue chains. * SC_TABLESIZE must be a power of two for SC_MASK to work properly. @@ -1034,6 +1036,120 @@ sleepq_abort(struct thread *td, int intrval) return (sleepq_resume_thread(sq, td, 0)); } +/* + * Prints the stacks of all threads presently sleeping on wchan/queue to + * the sbuf sb. Sets count_stacks_printed to the number of stacks actually + * printed. Typically, this will equal the number of threads sleeping on the + * queue, but may be less if sb overflowed before all stacks were printed. + */ +int +sleepq_sbuf_print_stacks(struct sbuf *sb, void *wchan, int queue, + int *count_stacks_printed) +{ + struct thread *td, *td_next; + struct sleepqueue *sq; + struct stack **st; + struct sbuf **td_infos; + int i, stack_idx, error, stacks_to_allocate; + bool finished, partial_print; + + error = 0; + finished = false; + partial_print = false; + + KASSERT(wchan != NULL, ("%s: invalid NULL wait channel", __func__)); + MPASS((queue >= 0) && (queue < NR_SLEEPQS)); + + stacks_to_allocate = 10; + for (i = 0; i < 3 && !finished ; i++) { + /* We cannot malloc while holding the queue's spinlock, so + * we do our mallocs now, and hope it is enough. If it + * isn't, we will free these, drop the lock, malloc more, + * and try again, up to a point. After that point we will + * give up and report ENOMEM. We also cannot write to sb + * during this time since the client may have set the + * SBUF_AUTOEXTEND flag on their sbuf, which could cause a + * malloc as we print to it. So we defer actually printing + * to sb until after we drop the spinlock. + */ + + /* Where we will store the stacks. */ + st = malloc(sizeof(struct stack *) * stacks_to_allocate, + M_TEMP, M_WAITOK); + for (stack_idx = 0; stack_idx < stacks_to_allocate; + stack_idx++) + st[stack_idx] = stack_create(); + + /* Where we will store the td name, tid, etc. */ + td_infos = malloc(sizeof(struct sbuf *) * stacks_to_allocate, + M_TEMP, M_WAITOK); + for (stack_idx = 0; stack_idx < stacks_to_allocate; + stack_idx++) + td_infos[stack_idx] = sbuf_new(NULL, NULL, + MAXCOMLEN + sizeof(struct thread *) * 2 + 40, + SBUF_FIXEDLEN); + + sleepq_lock(wchan); + sq = sleepq_lookup(wchan); + if (sq == NULL) { + /* This sleepq does not exist; exit and return ENOENT. */ + error = ENOENT; + finished = true; + sleepq_release(wchan); + goto loop_end; + } + + stack_idx = 0; + /* Save thread info */ + TAILQ_FOREACH_SAFE(td, &sq->sq_blocked[queue], td_slpq, + td_next) { + if (stack_idx >= stacks_to_allocate) + goto loop_end; + + /* Note the td_lock is equal to the sleepq_lock here. */ + stack_save_td(st[stack_idx], td); + + sbuf_printf(td_infos[stack_idx], "%d: %s %p", + td->td_tid, td->td_name, td); + + ++stack_idx; + } + + finished = true; + sleepq_release(wchan); + + /* Print the stacks */ + for (i = 0; i < stack_idx; i++) { + sbuf_finish(td_infos[i]); + sbuf_printf(sb, "--- thread %s: ---\n", sbuf_data(td_infos[i])); + stack_sbuf_print(sb, st[i]); + sbuf_printf(sb, "\n"); + + error = sbuf_error(sb); + if (error == 0) + *count_stacks_printed = stack_idx; + } + +loop_end: + if (!finished) + sleepq_release(wchan); + for (stack_idx = 0; stack_idx < stacks_to_allocate; + stack_idx++) + stack_destroy(st[stack_idx]); + for (stack_idx = 0; stack_idx < stacks_to_allocate; + stack_idx++) + sbuf_delete(td_infos[stack_idx]); + free(st, M_TEMP); + free(td_infos, M_TEMP); + stacks_to_allocate *= 10; + } + + if (!finished && error == 0) + error = ENOMEM; + + return (error); +} + #ifdef SLEEPQUEUE_PROFILING #define SLEEPQ_PROF_LOCATIONS 1024 #define SLEEPQ_SBUFSIZE 512 Index: /usr/src/sys/sys/fail.h =================================================================== --- /usr/src/sys/sys/fail.h (revision 296564) +++ /usr/src/sys/sys/fail.h (working copy) @@ -37,6 +37,11 @@ #include #include #include +#include +#include +#include +#include +#include /** * Failpoint return codes, used internally. @@ -49,7 +54,8 @@ enum fail_point_return_code { }; struct fail_point_entry; -TAILQ_HEAD(fail_point_entries, fail_point_entry); +struct fail_point_setting; + /** * Internal failpoint structure, tracking all the current details of the * failpoint. This structure is the core component shared between the @@ -57,22 +63,42 @@ struct fail_point_entry; * @ingroup failpoint_private */ struct fail_point { - const char *fp_name; /**< name of fail point */ - const char *fp_location; /**< file:line of fail point */ - struct fail_point_entries fp_entries; /**< list of entries */ + const char *fp_name; /* name of fail point */ + const char *fp_location; /* file:line of fail point */ + volatile int fp_ref_cnt; /** + * protects fp_setting: while holding + * a ref, fp_setting points to an + * unfreed fail_point_setting + */ + struct fail_point_setting * volatile fp_setting; int fp_flags; - void (*fp_sleep_fn)(void *); /**< Function to call at end of - * sleep for sleep failpoints */ - void *fp_sleep_arg; /**< Arg for sleep_fn */ + + /**< Function to call before sleep or pause */ + void (*fp_pre_sleep_fn)(void *); + /**< Arg for fp_pre_sleep_fn */ + void *fp_pre_sleep_arg; + + /**< Function to call after waking from sleep or pause */ + void (*fp_post_sleep_fn)(void *); + /**< Arg for fp_post_sleep_fn */ + void *fp_post_sleep_arg; }; #define FAIL_POINT_DYNAMIC_NAME 0x01 /**< Must free name on destroy */ +/**< Use timeout path for sleep instead of msleep */ +#define FAIL_POINT_USE_TIMEOUT_PATH 0x02 +/**< If fail point is set to sleep, replace the sleep call with delay */ +#define FAIL_POINT_NONSLEEPABLE 0x04 +#define FAIL_POINT_CV_DESC "fp cv no iterators" +#define FAIL_POINT_IS_OFF(fp) (__predict_true((fp)->fp_setting == NULL) || \ + __predict_true(fail_point_is_off(fp))) + __BEGIN_DECLS /* Private failpoint eval function -- use fail_point_eval() instead. */ enum fail_point_return_code fail_point_eval_nontrivial(struct fail_point *, - int *ret); + int *ret); /** * @addtogroup failpoint @@ -86,29 +112,65 @@ enum fail_point_return_code fail_point_eval_nontri void fail_point_init(struct fail_point *, const char *fmt, ...) __printflike(2, 3); +/* Return true iff this fail point is set to off, false otherwise */ +bool fail_point_is_off(struct fail_point *fp); + /** - * Set the sleep function for a fail point - * If sleep_fn is specified, then FAIL_POINT_SLEEP will result in a - * (*fp->sleep_fn)(fp->sleep_arg) call by the timer thread. Otherwise, - * if sleep_fn is NULL (default), then FAIL_POINT_SLEEP will result in the - * fail_point_eval() call sleeping. + * Set the pre-sleep function for a fail point + * If fp_post_sleep_fn is specified, then FAIL_POINT_SLEEP will result in a + * (*fp->fp_pre_sleep_fn)(fp->fp_pre_sleep_arg) call by the thread. */ -static __inline void -fail_point_sleep_set_func(struct fail_point *fp, void (*sleep_fn)(void *)) +static inline void +fail_point_sleep_set_pre_func(struct fail_point *fp, void (*sleep_fn)(void *)) { - fp->fp_sleep_fn = sleep_fn; + fp->fp_pre_sleep_fn = sleep_fn; } +static inline void +fail_point_sleep_set_pre_arg(struct fail_point *fp, void *sleep_arg) +{ + fp->fp_pre_sleep_arg = sleep_arg; +} + /** - * Set the argument for the sleep function for a fail point + * Set the post-sleep function. This will be passed to timeout if we take + * the timeout path. This must be set if you sleep using the timeout path. */ -static __inline void -fail_point_sleep_set_arg(struct fail_point *fp, void *sleep_arg) +static inline void +fail_point_sleep_set_post_func(struct fail_point *fp, void (*sleep_fn)(void *)) { - fp->fp_sleep_arg = sleep_arg; + fp->fp_post_sleep_fn = sleep_fn; } +static inline void +fail_point_sleep_set_post_arg(struct fail_point *fp, void *sleep_arg) +{ + fp->fp_post_sleep_arg = sleep_arg; +} /** + * If the FAIL_POINT_USE_TIMEOUT flag is set on a failpoint, then + * FAIL_POINT_SLEEP will result in a call to timeout instead of + * msleep. Note that if you sleep while this flag is set, you must + * set fp_post_sleep_fn or an error will occur upon waking. + */ +static inline void +fail_point_use_timeout_path(struct fail_point *fp, bool use_timeout, + void (*post_sleep_fn)(void *)) +{ + KASSERT(!use_timeout || post_sleep_fn != NULL || + (post_sleep_fn == NULL && fp->fp_post_sleep_fn != NULL), + ("Setting fp to use timeout, but not setting post_sleep_fn\n")); + + if (use_timeout) + fp->fp_flags |= FAIL_POINT_USE_TIMEOUT_PATH; + else + fp->fp_flags &= ~FAIL_POINT_USE_TIMEOUT_PATH; + + if (post_sleep_fn != NULL) + fp->fp_post_sleep_fn = post_sleep_fn; +} + +/** * Free the resources used by a fail-point. Pair with fail_point_init(). */ void fail_point_destroy(struct fail_point *); @@ -116,12 +178,11 @@ void fail_point_destroy(struct fail_point *); /** * Evaluate a failpoint. */ -static __inline enum fail_point_return_code +static inline enum fail_point_return_code fail_point_eval(struct fail_point *fp, int *ret) { - if (TAILQ_EMPTY(&fp->fp_entries)) { + if (__predict_true(fp->fp_setting == NULL)) return (FAIL_POINT_RC_CONTINUE); - } return (fail_point_eval_nontrivial(fp, ret)); } @@ -128,14 +189,45 @@ fail_point_eval(struct fail_point *fp, int *ret) __END_DECLS /* Declare a fail_point and its sysctl in a function. */ -#define _FAIL_POINT_NAME(name) _fail_point_##name -#define _FAIL_POINT_LOCATION() "(" __FILE__ ":" __XSTRING(__LINE__) ")" +#define _FAIL_POINT_NAME(name) _fail_point_##name +#define _FAIL_POINT_LOCATION() "(" __FILE__ ":" __XSTRING(__LINE__) ")" +#define _FAIL_POINT_INIT(parent, name, flags) \ + static struct fail_point _FAIL_POINT_NAME(name) = { \ + .fp_name = #name, \ + .fp_location = _FAIL_POINT_LOCATION(), \ + .fp_ref_cnt = 0, \ + .fp_setting = NULL, \ + .fp_flags = (flags), \ + .fp_pre_sleep_fn = NULL, \ + .fp_pre_sleep_arg = NULL, \ + .fp_post_sleep_fn = NULL, \ + .fp_post_sleep_arg = NULL, \ + }; \ + SYSCTL_OID(parent, OID_AUTO, name, \ + CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, \ + &_FAIL_POINT_NAME(name), 0, fail_point_sysctl, \ + "A", ""); \ + SYSCTL_OID(parent, OID_AUTO, status_##name, \ + CTLTYPE_STRING | CTLFLAG_RD | CTLFLAG_MPSAFE, \ + &_FAIL_POINT_NAME(name), 0, \ + fail_point_sysctl_status, "A", ""); +#define _FAIL_POINT_EVAL(name, cond, code...) \ + int RETURN_VALUE; \ + \ + if (__predict_false(cond && \ + fail_point_eval(&_FAIL_POINT_NAME(name), &RETURN_VALUE))) { \ + \ + code; \ + \ + } + /** - * Instantiate a failpoint which returns "value" from the function when triggered. - * @param parent The parent sysctl under which to locate the sysctl + * Instantiate a failpoint which returns "RETURN_VALUE" from the function + * when triggered. + * @param parent The parent sysctl under which to locate the fp's sysctl * @param name The name of the failpoint in the sysctl tree (and printouts) - * @return Instantly returns the return("value") specified in the + * @return Instantly returns the RETURN_VALUE specified in the * failpoint, if triggered. */ #define KFAIL_POINT_RETURN(parent, name) \ @@ -142,7 +234,8 @@ __END_DECLS KFAIL_POINT_CODE(parent, name, return RETURN_VALUE) /** - * Instantiate a failpoint which returns (void) from the function when triggered. + * Instantiate a failpoint which returns (void) from the function when + * triggered. * @param parent The parent sysctl under which to locate the sysctl * @param name The name of the failpoint in the sysctl tree (and printouts) * @return Instantly returns void, if triggered in the failpoint. @@ -153,7 +246,8 @@ __END_DECLS /** * Instantiate a failpoint which sets an error when triggered. * @param parent The parent sysctl under which to locate the sysctl - * @param name The name of the failpoint in the sysctl tree (and printouts) + * @param name The name of the failpoint in the sysctl tree (and + * printouts) * @param error_var A variable to set to the failpoint's specified * return-value when triggered */ @@ -164,7 +258,8 @@ __END_DECLS * Instantiate a failpoint which sets an error and then goes to a * specified label in the function when triggered. * @param parent The parent sysctl under which to locate the sysctl - * @param name The name of the failpoint in the sysctl tree (and printouts) + * @param name The name of the failpoint in the sysctl tree (and + * printouts) * @param error_var A variable to set to the failpoint's specified * return-value when triggered * @param label The location to goto when triggered. @@ -173,10 +268,58 @@ __END_DECLS KFAIL_POINT_CODE(parent, name, (error_var) = RETURN_VALUE; goto label) /** + * Instantiate a failpoint which sets its pre- and post-sleep callback + * mechanisms. + * @param parent The parent sysctl under which to locate the sysctl + * @param name The name of the failpoint in the sysctl tree (and + * printouts) + * @param pre_func Function pointer to the pre-sleep function, which will be + * called directly before going to sleep. + * @param pre_arg Argument to the pre-sleep function + * @param post_func Function pointer to the pot-sleep function, which will be + * called directly before going to sleep. + * @param post_arg Argument to the post-sleep function + */ +#define KFAIL_POINT_SLEEP_CALLBACKS(parent, name, pre_func, pre_arg, \ + post_func, post_arg) \ + KFAIL_POINT_CODE_SLEEP_CALLBACKS(parent, name, pre_func, \ + pre_arg, post_func, post_arg, return RETURN_VALUE) + +/** + * Instantiate a failpoint which runs arbitrary code when triggered, and sets + * its pre- and post-sleep callback mechanisms + * @param parent The parent sysctl under which to locate the sysctl + * @param name The name of the failpoint in the sysctl tree (and + * printouts) + * @param pre_func Function pointer to the pre-sleep function, which will be + * called directly before going to sleep. + * @param pre_arg Argument to the pre-sleep function + * @param post_func Function pointer to the pot-sleep function, which will be + * called directly before going to sleep. + * @param post_arg Argument to the post-sleep function + * @param code The arbitrary code to run when triggered. Can reference + * "RETURN_VALUE" if desired to extract the specified + * user return-value when triggered. Note that this is + * implemented with a do-while loop so be careful of + * break and continue statements. + */ +#define KFAIL_POINT_CODE_SLEEP_CALLBACKS(parent, name, pre_func, pre_arg, \ + post_func, post_arg, code...) \ + do { \ + _FAIL_POINT_INIT(parent, name) \ + _FAIL_POINT_NAME(name).fp_pre_sleep_fn = pre_func; \ + _FAIL_POINT_NAME(name).fp_pre_sleep_arg = pre_arg; \ + _FAIL_POINT_NAME(name).fp_post_sleep_fn = post_func; \ + _FAIL_POINT_NAME(name).fp_post_sleep_arg = post_arg; \ + _FAIL_POINT_EVAL(name, true, code) \ + } while (0) + + +/** * Instantiate a failpoint which runs arbitrary code when triggered. * @param parent The parent sysctl under which to locate the sysctl * @param name The name of the failpoint in the sysctl tree - * (and printouts) + * (and printouts) * @param code The arbitrary code to run when triggered. Can reference * "RETURN_VALUE" if desired to extract the specified * user return-value when triggered. Note that this is @@ -183,30 +326,24 @@ __END_DECLS * implemented with a do-while loop so be careful of * break and continue statements. */ -#define KFAIL_POINT_CODE(parent, name, code) \ -do { \ - int RETURN_VALUE; \ - static struct fail_point _FAIL_POINT_NAME(name) = { \ - #name, \ - _FAIL_POINT_LOCATION(), \ - TAILQ_HEAD_INITIALIZER(_FAIL_POINT_NAME(name).fp_entries), \ - 0, \ - NULL, NULL, \ - }; \ - SYSCTL_OID(parent, OID_AUTO, name, \ - CTLTYPE_STRING | CTLFLAG_RW | CTLFLAG_MPSAFE, \ - &_FAIL_POINT_NAME(name), 0, fail_point_sysctl, \ - "A", ""); \ - \ - if (__predict_false( \ - fail_point_eval(&_FAIL_POINT_NAME(name), &RETURN_VALUE))) { \ - \ - code; \ - \ - } \ -} while (0) +#define KFAIL_POINT_CODE(parent, name, code...) \ + do { \ + _FAIL_POINT_INIT(parent, name, 0) \ + _FAIL_POINT_EVAL(name, true, code) \ + } while (0) +#define KFAIL_POINT_CODE_FLAGS(parent, name, flags, code...) \ + do { \ + _FAIL_POINT_INIT(parent, name, flags) \ + _FAIL_POINT_EVAL(name, true, code) \ + } while (0) +#define KFAIL_POINT_CODE_COND(parent, name, cond, flags, code...) \ + do { \ + _FAIL_POINT_INIT(parent, name, flags) \ + _FAIL_POINT_EVAL(name, cond, code) \ + } while (0) + /** * @} * (end group failpoint) @@ -214,6 +351,7 @@ __END_DECLS #ifdef _KERNEL int fail_point_sysctl(SYSCTL_HANDLER_ARGS); +int fail_point_sysctl_status(SYSCTL_HANDLER_ARGS); /* The fail point sysctl tree. */ SYSCTL_DECL(_debug_fail_point); Index: /usr/src/sys/sys/param.h =================================================================== --- /usr/src/sys/sys/param.h (revision 296564) +++ /usr/src/sys/sys/param.h (working copy) @@ -215,6 +215,7 @@ #define PRIMASK 0x0ff #define PCATCH 0x100 /* OR'd with pri for tsleep to check signals */ #define PDROP 0x200 /* OR'd with pri to stop re-entry of interlock mutex */ +#define PSCHEDFUZZING 0x0400 /* don't use sched_fuzz on this call */ #define NZERO 0 /* default "nice" */ Index: /usr/src/sys/sys/proc.h =================================================================== --- /usr/src/sys/sys/proc.h (revision 296564) +++ /usr/src/sys/sys/proc.h (working copy) @@ -56,6 +56,7 @@ #include #include #include +#include #ifndef _KERNEL #include /* For structs itimerval, timeval. */ #else @@ -721,6 +722,8 @@ struct proc { #define P_MAGIC 0xbeefface #ifdef _KERNEL +extern struct sysctl_oid SYSCTL_NAME_MANGLE(_kern, sched_fuzz); +extern bool sched_fuzz_switch_next[MAXCPU]; /* Types and flags for mi_switch(). */ #define SW_TYPE_MASK 0xff /* First 8 bits are switch type */ Index: /usr/src/sys/sys/sleepqueue.h =================================================================== --- /usr/src/sys/sys/sleepqueue.h (revision 296564) +++ /usr/src/sys/sys/sleepqueue.h (working copy) @@ -107,5 +107,9 @@ int sleepq_type(void *wchan); void sleepq_wait(void *wchan, int pri); int sleepq_wait_sig(void *wchan, int pri); +#include +int sleepq_sbuf_print_stacks(struct sbuf *sb, void *wchan, int queue, + int *count_stacks_printed); + #endif /* _KERNEL */ #endif /* !_SYS_SLEEPQUEUE_H_ */ Index: /usr/src/sys/sys/sysctl.h =================================================================== --- /usr/src/sys/sys/sysctl.h (revision 296564) +++ /usr/src/sys/sys/sysctl.h (working copy) @@ -265,15 +265,16 @@ TAILQ_HEAD(sysctl_ctx_list, sysctl_ctx_entry); }; \ DATA_SET(sysctl_set, id) +#define SYSCTL_NAME_MANGLE(parent, name) sysctl__##parent##_##name + /* This constructs a static "raw" MIB oid. */ -#define SYSCTL_OID(parent, nbr, name, kind, a1, a2, handler, fmt, descr) \ - static SYSCTL_OID_RAW(sysctl__##parent##_##name, \ - SYSCTL_CHILDREN(&sysctl__##parent), \ - nbr, #name, kind, a1, a2, handler, fmt, descr) +#define SYSCTL_OID(parent, nbr, name, kind, a1, a2, handler, fmt, descr) \ + static SYSCTL_OID_GLOBAL(parent, nbr, name, kind, a1, a2, handler, fmt,\ + descr) /* This constructs a global "raw" MIB oid. */ #define SYSCTL_OID_GLOBAL(parent, nbr, name, kind, a1, a2, handler, fmt, descr) \ - SYSCTL_OID_RAW(sysctl__##parent##_##name, \ + SYSCTL_OID_RAW(SYSCTL_NAME_MANGLE(parent, name), \ SYSCTL_CHILDREN(&sysctl__##parent), \ nbr, #name, kind, a1, a2, handler, fmt, descr) Index: /usr/src/sys/sys/systm.h =================================================================== --- /usr/src/sys/sys/systm.h (revision 296564) +++ /usr/src/sys/sys/systm.h (working copy) @@ -385,8 +385,13 @@ int msleep_spin_sbt(void *chan, struct mtx *mtx, c 0, C_HARDCLOCK) int pause_sbt(const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags); +int pause_sbt_pri(const char *wmesg, int priority, sbintime_t sbt, + sbintime_t pr, int flags); #define pause(wmesg, timo) \ pause_sbt((wmesg), tick_sbt * (timo), 0, C_HARDCLOCK) +#define pause_sched_fuzz(wmesg, timo) \ + pause_sbt_pri((wmesg), PSCHEDFUZZING, \ + tick_sbt * (timo), 0, C_HARDCLOCK) #define tsleep(chan, pri, wmesg, timo) \ _sleep((chan), NULL, (pri), (wmesg), tick_sbt * (timo), \ 0, C_HARDCLOCK) Index: /usr/src/sys/vm/uma_core.c =================================================================== --- /usr/src/sys/vm/uma_core.c (revision 296564) +++ /usr/src/sys/vm/uma_core.c (working copy) @@ -67,6 +67,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -2132,6 +2133,10 @@ uma_zalloc_arg(uma_zone_t zone, void *udata, int f /* This is unfortunate but should not be fatal. */ } #endif + + KFAIL_POINT_CODE_FLAGS(_kern_sched_fuzz, sched_fuzz_malloc, + FAIL_POINT_NONSLEEPABLE, {}); + /* * If possible, allocate from the per-CPU cache. There are two * requirements for safe access to the per-CPU cache: (1) the thread -- Test scenario: misc/schedfuzz.sh