GENERIC HEAD from 2011-11-14 18:00:15 UTC, r227502M, vmcore.222 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r227502M: Mon Nov 14 19:19:08 CET 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531710464 (3368 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 130138 free (978 frags, 16145 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 138014 free (4446 frags, 16696 blocks, 0.4% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 114668882 free (1370 frags, 14333439 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 1398147 free (287667 frags, 138810 blocks, 2.8% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 7019148 free (69452 frags, 868712 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xe0b208c0 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xc7e48200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f234e8,2e687361,38323a63,a34,f18257e8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a5647b,c0f26eef,c11ff078,11c,f1825878,...) at kdb_backtrace+0x2a _witness_debugger(c0f26eef,c7e48200,c0f56d26,c71693a0,c0f569b8,...) at _witness_debugger+0x25 witness_checkorder(c7e48200,9,c0f569af,11c,0,...) at witness_checkorder+0x86f _sx_xlock(c7e48200,0,c0f569af,11c,c78aa5e4,...) at _sx_xlock+0x85 ufsdirhash_acquire(e0b20860,c78aa5e4,f18259f4,e184ee1c,f1825948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c78aa5e4,f18259f4,2e1c,f1825934,f1825938,...) at ufsdirhash_add+0x13 ufs_direnter(c7be9828,c7e01828,f18259f4,f1825b84,e0b20ee0,...) at ufs_direnter+0x789 ufs_mkdir(f1825c14,c0f76648,0,0,f1825bac,...) at ufs_mkdir+0xa36 VOP_MKDIR_APV(c106a620,f1825c14,f1825b84,f1825bac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c79135c0,ffffff9c,28404020,0,1c0,...) at kern_mkdirat+0x291 kern_mkdir(c79135c0,28404020,0,1c0,f1825d1c,...) at kern_mkdir+0x2e sys_mkdir(c79135c0,f1825cec,c0f6f44a,c0f27aaa,202,...) at sys_mkdir+0x29 syscall(f1825d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28174583, esp = 0xbfbfe8cc, ebp = 0xbfbfed78 --- Clearing /tmp (X related). Starting mountd. NFSv4 is disabled Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xc7f39df0 ufs (ufs) @ kern/vfs_lookup.c:540 2nd 0xe0b278e0 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 3rd 0xc7f3946c ufs (ufs) @ kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f234e8,3a632e73,a303632,0,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a5647b,c0f26f08,c11feb30,877,f18ed344,...) at kdb_backtrace+0x2a _witness_debugger(c0f26f08,c7f3946c,c0f15c7c,c7169338,c0f2eca5,...) at _witness_debugger+0x25 witness_checkorder(c7f3946c,9,c0f2ec9c,877,0,...) at witness_checkorder+0x86f __lockmgr_args(c7f3946c,80100,c7f394d8,0,0,...) at __lockmgr_args+0x834 ffs_lock(f18ed468,60,0,80100,c7f39414,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c106a620,f18ed468,c71621e0,c1079680,c7f39414,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7f39414,80100,c0f2ec9c,877,4,...) at _vn_lock+0x78 vget(c7f39414,80100,c790f2e0,50,0,...) at vget+0xbb vfs_hash_get(c7be0000,160525,80000,c790f2e0,f18ed5ac,...) at vfs_hash_get+0xed ffs_vgetf(c7be0000,160525,80000,f18ed5ac,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c7f39d98,e0b27880,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(c7f39d98,1,c11feb20,c7f39e90,c7165370,...) at ffs_syncvnode+0x24c ffs_truncate(c7f39d98,200,0,880,c71a5300,...) at ffs_truncate+0x817 ufs_direnter(c7f39d98,c7f39414,f18ed918,f18edbc0,0,...) at ufs_direnter+0x971 ufs_makeinode(f18edbc0,c106aba0,f18edb04,f18eda60,c0d7aa65,...) at ufs_makeinode+0x635 ufs_create(f18edb04,c0f7680e,0,0,f18edb80,...) at ufs_create+0x30 VOP_CREATE_APV(c106a620,f18edb04,f18edbc0,f18eda9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(f18edb80,f18edc48,1b0,0,c71a5300,...) at vn_open_cred+0x205 vn_open(f18edb80,f18edc48,1b0,c7d6f1c0,4ec16cb4,...) at vn_open+0x3b kern_openat(c790f2e0,ffffff9c,28839f60,0,a03,...) at kern_openat+0x1ec kern_open(c790f2e0,28839f60,0,a02,1b0,...) at kern_open+0x35 sys_open(c790f2e0,f18edcec,c0f6f44a,c0f27aaa,216,...) at sys_open+0x30 syscall(f18edd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283d65c3, esp = 0xbfbfc4fc, ebp = 0xbfbfc588 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc797f310 ufs (ufs) @ kern/vfs_mount.c:1209 2nd 0xc792ab38 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1834 KDB: stack backtrace: db_trace_self_wrapper(c0f234e8,38313a63,a3433,c0a6878c,8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a5647b,c0f26eef,c11fe700,72a,f192ea10,...) at kdb_backtrace+0x2a _witness_debugger(c0f26eef,c792ab38,c0f0dae7,c7169198,c0f52eb9,...) at _witness_debugger+0x25 witness_checkorder(c792ab38,9,c0f52eb0,72a,c792aba4,...) at witness_checkorder+0x86f __lockmgr_args(c792ab38,80400,c792aba4,0,0,...) at __lockmgr_args+0x834 vop_stdlock(f192eb30,4,c0f1da4b,80400,c792aae0,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c1040680,f192eb30,c1232700,c1079680,c792aae0,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c792aae0,80400,c0f52eb0,72a,c7be05a8,...) at _vn_lock+0x78 softdep_flushworklist(c7be05a8,f192ebdc,c7bbd8a0,5dc,0,...) at softdep_flushworklist+0x47 ffs_sync(c7be05a8,1,f192ec30,4f2,c7025000,...) at ffs_sync+0x2fd dounmount(c7be05a8,8080000,c7bbd8a0,483,5b85c5ea,...) at dounmount+0x3f8 sys_unmount(c7bbd8a0,f192ecec,c,c,206,...) at sys_unmount+0x331 syscall(f192ed28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280debdb, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes Nov 14 20:34:19 x4 su: pho to root on /dev/pts/0 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 25632 files, 3645099 used, 114668882 free (1370 frags, 14333439 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 25632 files, 3645099 used, 114668882 free (1370 frags, 14333439 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Mon Nov 14 20:36:21 CET 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Nov 14 20:38:08 x4 su: pho to root on /dev/pts/1 FreeBSD/i386 (x4.osted.lan) (console) login: vp = 0xc9f5c2b8, unref'ed lowervp c71c34f0 c7398844 c8d4015c c9f5c2b8 deadc0de deadc0de deadc0de c1051060 panic: null with unref'ed lowervp cpuid = 1 KDB: enter: panic [ thread pid 170 tid 100240 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> show mount 0xc789cb50 /dev/ad4s1a on / (ufs) 0xc789d000 devfs on /dev (devfs) 0xc789c87c /dev/ad4s1f on /home (ufs) 0xc7be02d4 /dev/ad4s1d on /usr (ufs) 0xc7be0000 /dev/ad4s1e on /var (ufs) 0xc789db50 procfs on /proc (procfs) 0xc789d87c linprocfs on /usr/compat/linux/proc (linprocfs) 0xc789c5a8 /dev/label/tmp on /tmp (ufs) 0xc84445a8 /bin on /mnt (nullfs) More info: show mount db> show mount 0xc84445a8 0xc84445a8 /bin on /mnt (nullfs) mnt_flag = EXPORTED, LOCAL mnt_kern_flag = MPSAFE mnt_opt = target, fstype, fspath, from, errmsg mnt_stat = { version=537068824 type=53 flags=0x0000000000005000 bsize=2048 iosize=16384 blocks=1013015 bfree=130138 bavail=49097 files=282622 ffree=273783 syncwrites=0 asyncwrites=0 syncreads=0 asyncreads=0 namemax=255 owner=0 fsid=[695730169, 41] } mnt_cred = { uid=0 ruid=0 } mnt_ref = 3 mnt_gen = 321 mnt_nvnodelistsize = 3 mnt_writeopcount = 0 mnt_noasync = 0 mnt_maxsymlinklen = 0 mnt_iosize_max = 65536 mnt_hashseed = 3404481644 mnt_secondary_writes = 0 mnt_secondary_accwrites = 0 mnt_gjprovider = NULL vnode 0xc9f55ae0: tag null, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VV_ROOT) lock type ufs: UNLOCKED #0 0xc0a055fe at __lockmgr_args+0xc1e #1 0xc0c63451 at ffs_lock+0xa1 #2 0xc0d79a95 at VOP_LOCK1_APV+0xb5 #3 0xc0ac81a8 at _vn_lock+0x78 #4 0xc8407137 at null_vptocnp+0x127 #5 0xc0d77be6 at VOP_VPTOCNP_APV+0xd6 #6 0xc0aa226c at vn_vptocnp_locked+0x27c #7 0xc0aa242c at vn_fullpath1+0x7c #8 0xc0aa2b76 at vn_fullpath+0xb6 #9 0xc09a4e13 at procfs_doprocfile+0x83 #10 0xc09aa943 at pfs_readlink+0x423 #11 0xc0d78865 at VOP_READLINK_APV+0xc5 #12 0xc0abe466 at kern_readlinkat+0x1a6 #13 0xc0abe4fc at kern_readlink+0x3c #14 0xc0abe538 at sys_readlink+0x38 #15 0xc0d56514 at syscall+0x2a4 #16 0xc0d3f941 at Xint0x80_syscall+0x21 vp=0xc9f55ae0, lowervp=0xc765715c vnode 0xca2ea828: tag syncer, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type syncer: UNLOCKED #0 0xc0a055fe at __lockmgr_args+0xc1e #1 0xc0aa6785 at vop_stdlock+0x65 #2 0xc0d79a95 at VOP_LOCK1_APV+0xb5 #3 0xc0ac81a8 at _vn_lock+0x78 #4 0xc0abc86c at vfs_allocate_syncvnode+0x6c #5 0xc0ab03c1 at vfs_donmount+0xe81 #6 0xc0ab0e94 at sys_nmount+0x84 #7 0xc0d56514 at syscall+0x2a4 #8 0xc0d3f941 at Xint0x80_syscall+0x21 vnode 0xc9f5c2b8: tag null, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VV_TEXT|VI_OWEINACT) VI_LOCKed v_object 0xc8ae02a8 ref 0 pages 6 lock type ufs: UNLOCKED #0 0xc0a055fe at __lockmgr_args+0xc1e #1 0xc0c63451 at ffs_lock+0xa1 #2 0xc0d79a95 at VOP_LOCK1_APV+0xb5 #3 0xc0ac81a8 at _vn_lock+0x78 #4 0xc0c932b0 at vm_object_deallocate+0x220 #5 0xc0c88e91 at vm_map_entry_deallocate+0x21 #6 0xc0c88efa at vm_map_process_deferred+0x2a #7 0xc0c8a09a at _vm_map_unlock+0x4a #8 0xc0c8a70b at vm_map_remove+0x6b #9 0xc0c8dae3 at vmspace_exit+0xc3 #10 0xc09eebce at exit1+0x60e #11 0xc09ef63d at sys_sys_exit+0x1d #12 0xc0d56514 at syscall+0x2a4 #13 0xc0d3f941 at Xint0x80_syscall+0x21 panic: _mtx_lock_sleep: recursed on non-recursive mutex vnode interlock @ /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:299 cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper(c0f234e8,0,28,0,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c10c3180,0,c0f200e9,17b,c84a7b80,...) at kdb_backtrace+0x2a mi_switch(100,0,c0f21562,984,1,...) at mi_switch+0x10d sched_bind(c84a7b80,0,c0f1f237,129,ffffffff,...) at sched_bind+0xb9 kern_reboot(4,c0f1f4c8,c7964000,c0d3f941,1,...) at kern_reboot+0x40 panic(c0f1dc91,c0f270e9,c840724a,12b,c8d40220,...) at panic+0x151 _mtx_lock_sleep(c8d40220,c84a7b80,10,c840724a,12b,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c8d40220,10,c840724a,12b,c9f5c2b8,...) at _mtx_lock_flags+0xf7 null_checkvp(c9f5c2b8,c84073ef,2dc,f1ad6494,f1ad6478,...) at null_checkvp+0x3a null_print(f1ad6494,c0f13423,c9f5c2b8,c9f5c2b8,f1ad65bc,...) at null_print+0x25 VOP_PRINT_APV(c8408700,f1ad6494,0,6,0,...) at VOP_PRINT_APV+0xa5 vn_printf(c9f5c2b8,c0f2f581,0,5000,0,...) at vn_printf+0x47d db_show_mount(c84445a8,1,ffffffff,f1ad68f8,1,...) at db_show_mount+0x1736 db_command(c0d94813,f1ad69e0,c04e54ed,c0f1f4c8,fb,...) at db_command+0x381 db_command_loop(c0f1f4c8,fb,3,0,0,...) at db_command_loop+0x5a db_trap(3,0,46,1,d,...) at db_trap+0xdd kdb_trap(3,0,f1ad6aa8,f1ad6a28,c84a3000,...) at kdb_trap+0xb3 trap(f1ad6aa8) at trap+0x66b calltrap() at calltrap+0x6 --- trap 0x3, eip = 0xc0a51b2a, esp = 0xf1ad6ae8, ebp = 0xf1ad6af0 --- kdb_enter(c0f1f4c8,c0f1f4c8,c840729d,f1ad6b28,1,...) at kdb_enter+0x3a panic(c840729d,c1051060,c840724a,12b,c8408700,...) at panic+0x134 null_checkvp(c9f5c2b8,c84073ef,246,246,80100,...) at null_checkvp+0x108 null_lock(f1ad6ba8,c0a6659a,c0a0d4f3,80100,c9f5c2b8,...) at null_lock+0x41 VOP_LOCK1_APV(c8408700,f1ad6ba8,c0a0d923,c1079680,c9f5c2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c9f5c2b8,80100,c0f2ec9c,8e7,c10b47a4,...) at _vn_lock+0x78 vputx(f1ad6c5c,c09ef408,c9f5c2b8,c0f19b31,160,...) at vputx+0x266 vrele(c9f5c2b8,c0f19b31,160,131,281ed6f0,...) at vrele+0x10 exit1(c84a7b80,0,f1ad6d1c,c0d56514,c84a7b80,...) at exit1+0xe48 sys_sys_exit(c84a7b80,f1ad6cec,c0f6f44a,c0eedf77,286,...) at sys_sys_exit+0x1d syscall(f1ad6d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x2815d84f, esp = 0xbfbfe21c, ebp = 0xbfbfe228 --- db> run pho db:0:pho> bt Tracing pid 170 tid 100240 td 0xc84a7b80 kdb_enter(c0f1f4c8,c0f1f4c8,c840729d,f1ad6b28,1,...) at kdb_enter+0x3a panic(c840729d,c1051060,c840724a,12b,c8408700,...) at panic+0x134 null_checkvp(c9f5c2b8,c84073ef,246,246,80100,...) at null_checkvp+0x108 null_lock(f1ad6ba8,c0a6659a,c0a0d4f3,80100,c9f5c2b8,...) at null_lock+0x41 VOP_LOCK1_APV(c8408700,f1ad6ba8,c0a0d923,c1079680,c9f5c2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c9f5c2b8,80100,c0f2ec9c,8e7,c10b47a4,...) at _vn_lock+0x78 vputx(f1ad6c5c,c09ef408,c9f5c2b8,c0f19b31,160,...) at vputx+0x266 vrele(c9f5c2b8,c0f19b31,160,131,281ed6f0,...) at vrele+0x10 exit1(c84a7b80,0,f1ad6d1c,c0d56514,c84a7b80,...) at exit1+0xe48 sys_sys_exit(c84a7b80,f1ad6cec,c0f6f44a,c0eedf77,286,...) at sys_sys_exit+0x1d syscall(f1ad6d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x2815d84f, esp = 0xbfbfe21c, ebp = 0xbfbfe228 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x7ab200 curthread = 0xc71c88a0: pid 11 "idle: cpu0" curpcb = 0xc6f3fd80 fpcurthread = none idlethread = 0xc71c88a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5eb5200 curthread = 0xc84a7b80: pid 170 "ls" curpcb = 0xf1ad6d80 fpcurthread = 0xc84a7b80: pid 170 "ls" idlethread = 0xc71c85c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5eb8200 curthread = 0xc71c82e0: pid 11 "idle: cpu2" curpcb = 0xc6f45d80 fpcurthread = none idlethread = 0xc71c82e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5ebb200 curthread = 0xc71c8000: pid 11 "idle: cpu3" curpcb = 0xc6f48d80 fpcurthread = none idlethread = 0xc71c8000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 170 (ls) thread 0xc84a7b80 (100240) exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xc8d40220) locked @ /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:299 exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xc9f5c37c) locked @ kern/vfs_subr.c:2246 db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc789cb50 /dev/ad4s1a on / (ufs) 0xc789d000 devfs on /dev (devfs) 0xc789c87c /dev/ad4s1f on /home (ufs) 0xc7be02d4 /dev/ad4s1d on /usr (ufs) 0xc7be0000 /dev/ad4s1e on /var (ufs) 0xc789db50 procfs on /proc (procfs) 0xc789d87c linprocfs on /usr/compat/linux/proc (linprocfs) 0xc789c5a8 /dev/label/tmp on /tmp (ufs) 0xc84445a8 /bin on /mnt (nullfs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 170 169 51255 0 RE+ CPU 1 ls 169 157 51255 0 S+ wait 0xc87122c4 sh 157 51255 51255 0 S+ piperd 0xc82044b0 sh 99942 49100 49097 1001 S nanslp 0xc10b5ea4 sleep 51255 2273 51255 0 S+ wait 0xc83f42c4 sh 50777 2162 50777 0 S+ select 0xc81d0ca4 vim 49101 49097 49097 1001 S piperd 0xc78d3c80 awk 49100 49097 49097 1001 S wait 0xc9da584c sh 49099 49095 49099 1001 Ss+ select 0xcb206c24 top 49098 49096 49098 1001 Ss kqread 0xc787da00 tail 49097 49094 49097 1001 SWs wait 0xc7fc0588 sh 49096 49062 49062 1001 S select 0xc92cf764 sshd 49095 49061 49061 1001 S select 0xcb3ac524 sshd 49094 49060 49060 1001 S select 0xcb7950e4 sshd 49062 2090 49062 0 Ss select 0xc73987e4 sshd 49061 2090 49061 0 Ss select 0xc888f464 sshd 49060 2090 49060 0 Ss select 0xcb5e5464 sshd 9867 9853 9867 1001 Rs+ bash 9853 9576 9576 1001 S select 0xc7d5e764 sshd 9576 2090 9576 0 Ss select 0xc7398824 sshd 2273 2269 2273 0 SW+ wait 0xc790e000 bash 2269 2268 2269 0 SW+ pause 0xc793b8a4 csh 2268 2263 2268 1001 SW+ wait 0xc7fc0b10 su 2263 2262 2263 1001 SWs+ wait 0xc8207b10 bash 2262 2260 2260 1001 S select 0xc73989e4 sshd 2260 2090 2260 0 Ss select 0xc81610a4 sshd 2251 1 2251 0 Ss+ ttyin 0xc72e9870 getty 2250 1 2250 0 Ss+ ttyin 0xc72e9a70 getty 2249 1 2249 0 Ss+ ttyin 0xc72e9c70 getty 2248 1 2248 0 Ss+ ttyin 0xc72e9e70 getty 2247 1 2247 0 Ss+ ttyin 0xc757e070 getty 2246 1 2246 0 Ss+ ttyin 0xc757e270 getty 2245 1 2245 0 Ss+ ttyin 0xc757e470 getty 2244 1 2244 0 Ss+ ttyin 0xc72e8670 getty 2243 1 2243 0 Ss+ ttyin 0xc72e8470 getty 2203 1 2203 0 Ss select 0xc7398a24 inetd 2162 2158 2162 0 SW+ wait 0xc8206b10 bash 2158 2157 2158 0 SW+ pause 0xc820731c csh 2157 2152 2157 1001 SW+ wait 0xc793e84c su 2152 2151 2152 1001 SWs+ wait 0xc7c5f000 bash 2151 2149 2149 1001 S select 0xc7398ca4 sshd 2149 2090 2149 0 Ss select 0xc8161264 sshd 2110 1 2110 0 Ss nanslp 0xc10b5ea4 cron 2102 1 2102 25 SWs pause 0xc7c60058 sendmail 2098 1 2098 0 Ss select 0xc7397e24 sendmail 2090 1 2090 0 Ss select 0xc73973a4 sshd 2030 1 2030 0 Ss select 0xc795b2a4 moused 1952 1 1952 0 Ss select 0xc7398ce4 ntpd 1852 1845 1845 0 S (threaded) nfsd 100149 S rpcsvc 0xc796ab90 nfsd: service 100148 S rpcsvc 0xc7891250 nfsd: service 100147 S rpcsvc 0xc796abd0 nfsd: service 100080 S rpcsvc 0xc7891290 nfsd: master 1845 1 1845 0 Ss select 0xc7ed1364 nfsd 1835 1 1835 0 Ss select 0xc795b6a4 mountd 1736 1 1736 0 Ss select 0xc7d5eca4 rpcbind 1709 1 1709 0 Ss select 0xc7397ce4 syslogd 1502 1 1502 0 Ss select 0xc7398024 devd 18 0 0 0 DL sdflush 0xc1232740 [softdepflush] 17 0 0 0 DL syncer 0xc122c074 [syncer] 16 0 0 0 DL vlruwt 0xc77ef84c [vnlru] 9 0 0 0 DL psleep 0xc122bd08 [bufdaemon] 8 0 0 0 DL pgzero 0xc12345dc [pagezero] 7 0 0 0 DL psleep 0xc12341c8 [vmdaemon] 6 0 0 0 DL psleep 0xc1234190 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc107ab54 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc122db78 [sctp_iterator] 3 0 0 0 DL - 0xc73eea3c [fdc0] 2 0 0 0 SL - 0xc747c000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc745cd34 [usbus5] 100057 D - 0xc745cd04 [usbus5] 100056 D - 0xc745ccd4 [usbus5] 100055 D - 0xc745cca4 [usbus5] 100053 D - 0xc7451b5c [usbus4] 100052 D - 0xc7451b2c [usbus4] 100051 D - 0xc7451afc [usbus4] 100050 D - 0xc7451acc [usbus4] 100049 D - 0xc744db5c [usbus3] 100048 D - 0xc744db2c [usbus3] 100047 D - 0xc744dafc [usbus3] 100046 D - 0xc744dacc [usbus3] 100045 D - 0xc7443b5c [usbus2] 100044 D - 0xc7443b2c [usbus2] 100043 D - 0xc7443afc [usbus2] 100042 D - 0xc7443acc [usbus2] 100040 D - 0xc743bb5c [usbus1] 100039 D - 0xc743bb2c [usbus1] 100038 D - 0xc743bafc [usbus1] 100037 D - 0xc743bacc [usbus1] 100035 D - 0xc7428b5c [usbus0] 100034 D - 0xc7428b2c [usbus0] 100033 D - 0xc7428afc [usbus0] 100032 D - 0xc7428acc [usbus0] 14 0 0 0 DL - 0xc10b4ca4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc10b28a8 [g_down] 100014 D - 0xc10b28a4 [g_up] 100013 D - 0xc10b289c [g_event] 12 0 0 0 WL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100027 I [swi6: Giant taskq] 100025 I [swi5: +] 100024 I [swi2: cambio] 100018 I [swi6: task queue] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc71c6b10 [init] 10 0 0 0 DL audit_wo 0xc1231e80 [audit] 0 0 0 0 DLs (threaded) [kernel] 100077 D - 0xc77cfb00 [mca taskq] 100067 D - 0xc10b4ca4 [deadlkres] 100061 D - 0xc747b140 [fw0_taskq] 100026 D - 0xc7354400 [thread taskq] 100023 D - 0xc7354480 [kqueue taskq] 100022 D - 0xc73544c0 [acpi_task_2] 100021 D - 0xc73544c0 [acpi_task_1] 100020 D - 0xc73544c0 [acpi_task_0] 100019 D - 0xc7354500 [ffs_trim taskq] 100016 D - 0xc7195c40 [firmware taskq] 100000 D sched 0xc10b2980 [swapper] 99968 9867 99968 1001 Z+ umtx db:0:ps> allt Tracing command ls pid 170 tid 100240 td 0xc84a7b80 kdb_enter(c0f1f4c8,c0f1f4c8,c840729d,f1ad6b28,1,...) at kdb_enter+0x3a panic(c840729d,c1051060,c840724a,12b,c8408700,...) at panic+0x134 null_checkvp(c9f5c2b8,c84073ef,246,246,80100,...) at null_checkvp+0x108 null_lock(f1ad6ba8,c0a6659a,c0a0d4f3,80100,c9f5c2b8,...) at null_lock+0x41 VOP_LOCK1_APV(c8408700,f1ad6ba8,c0a0d923,c1079680,c9f5c2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c9f5c2b8,80100,c0f2ec9c,8e7,c10b47a4,...) at _vn_lock+0x78 vputx(f1ad6c5c,c09ef408,c9f5c2b8,c0f19b31,160,...) at vputx+0x266 vrele(c9f5c2b8,c0f19b31,160,131,281ed6f0,...) at vrele+0x10 exit1(c84a7b80,0,f1ad6d1c,c0d56514,c84a7b80,...) at exit1+0xe48 sys_sys_exit(c84a7b80,f1ad6cec,c0f6f44a,c0eedf77,286,...) at sys_sys_exit+0x1d syscall(f1ad6d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x2815d84f, esp = 0xbfbfe21c, ebp = 0xbfbfe228 --- Tracing command sh pid 169 tid 100277 td 0xc871a8a0 sched_switch(c871a8a0,0,104,191,8096e1ce,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c871a8a0,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c87122c4,6c,c0f27c03,100,0,...) at sleepq_wait_sig+0x17 _sleep(c87122c4,c871234c,16c,c0f27c03,0,...) at _sleep+0x35c kern_wait(c871a8a0,ffffffff,f1b45c60,2,0,...) at kern_wait+0x626 sys_wait4(c871a8a0,f1b45cec,c0f6f44a,c0f27a3a,216,...) at sys_wait4+0x3b syscall(f1b45d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2816c7eb, esp = 0xbfbfe5bc, ebp = 0xbfbfe5d8 --- Tracing command sh pid 157 tid 100180 td 0xc7fc58a0 sched_switch(c7fc58a0,0,104,191,7f16ac0b,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7fc58a0,0,c0f24e20,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c82044b0,5c,c0f2763b,100,0,...) at sleepq_wait_sig+0x17 _sleep(c82044b0,c8204628,15c,c0f2763b,0,...) at _sleep+0x35c pipe_read(c78ccd58,f19b8c44,c7932680,0,c7fc58a0,...) at pipe_read+0x417 dofileread(f19b8c44,ffffffff,ffffffff,0,f19b8c20,...) at dofileread+0x9e kern_readv(c7fc58a0,3,f19b8c44,f19b8c64,1,...) at kern_readv+0x68 sys_read(c7fc58a0,f19b8cec,c,c,206,...) at sys_read+0x4f syscall(f19b8d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281fd563, esp = 0xbfbfe71c, ebp = 0xbfbfe858 --- Tracing command sleep pid 99942 tid 100289 td 0xc87108a0 sched_switch(c87108a0,0,104,191,3788e899,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c87108a0,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(2711,c0a5cc10,c87108a0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10b5ea4,6c,c0f209f7,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10b5ea4,0,16c,c0f209f7,2711,...) at _sleep+0x328 kern_nanosleep(c87108a0,f1b69c50,f1b69c58,a,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c87108a0,f1b69cec,c0f6f44a,c0f28458,246,...) at sys_nanosleep+0x6f syscall(f1b69d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x28167a47, esp = 0xbfbfec5c, ebp = 0xbfbfeca8 --- Tracing command sh pid 51255 tid 100208 td 0xc8401b80 sched_switch(c8401b80,0,104,191,6371ead3,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c8401b80,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c83f42c4,6c,c0f27c03,100,0,...) at sleepq_wait_sig+0x17 _sleep(c83f42c4,c83f434c,16c,c0f27c03,0,...) at _sleep+0x35c kern_wait(c8401b80,ffffffff,f1a1cc60,2,0,...) at kern_wait+0x626 sys_wait4(c8401b80,f1a1ccec,c0f6f44a,c0f27be2,216,...) at sys_wait4+0x3b syscall(f1a1cd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2816c7eb, esp = 0xbfbfe68c, ebp = 0xbfbfe6a8 --- Tracing command vim pid 50777 tid 100292 td 0xcb2648a0 sched_switch(cb2648a0,0,104,191,9d9a0715,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(cb2648a0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c81d0c90,0,c0f1da4b,cb2648a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c81d0ca4,0,f1b72a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c81d0ca4,c81d0c90,c0f2743e,6a4,c7d82000,...) at _cv_wait_sig+0x243 seltdwait(c7d82000,d8,c7ebea00,cb2648a0,c9b49058,...) at seltdwait+0xa2 kern_select(cb2648a0,1,bfbfe064,0,bfbfdfe4,0,20,c88172c4,28f9c138) at kern_select+0x587 sys_select(cb2648a0,f1b72cec,c0f6f44a,c0f28b20,286,...) at sys_select+0x66 syscall(f1b72d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x290994e3, esp = 0xbfbfdf7c, ebp = 0xbfbfdfa8 --- Tracing command awk pid 49101 tid 100300 td 0xcb264000 sched_switch(cb264000,0,104,191,375d306a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(cb264000,0,c0f24e20,1a7,5c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c78d3c80,5c,c0f2763b,100,0,...) at sleepq_wait_sig+0x17 _sleep(c78d3c80,c78d3df8,15c,c0f2763b,0,...) at _sleep+0x35c pipe_read(c78b6cb0,f1b8ac44,c877ba80,0,cb264000,...) at pipe_read+0x417 dofileread(f1b8ac44,ffffffff,ffffffff,0,f1b8ac20,...) at dofileread+0x9e kern_readv(cb264000,0,f1b8ac44,f1b8ac64,1,...) at kern_readv+0x68 sys_read(cb264000,f1b8acec,c0f6f44a,f1b8ad80,216,...) at sys_read+0x4f syscall(f1b8ad28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281bc563, esp = 0xbfbfe93c, ebp = 0xbfbfe958 --- Tracing command sh pid 49100 tid 100298 td 0xcb3a38a0 sched_switch(cb3a38a0,0,104,191,37688c7f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(cb3a38a0,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c9da584c,6c,c0f27c03,100,0,...) at sleepq_wait_sig+0x17 _sleep(c9da584c,c9da58d4,16c,c0f27c03,0,...) at _sleep+0x35c kern_wait(cb3a38a0,ffffffff,f1b84c60,2,0,...) at kern_wait+0x626 sys_wait4(cb3a38a0,f1b84cec,c0f6f44a,c0f27be2,216,...) at sys_wait4+0x3b syscall(f1b84d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2816c7eb, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command top pid 49099 tid 100199 td 0xc7d70000 sched_switch(c7d70000,0,104,191,3dd7f5e7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d70000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f19fba28,c0a0d57a,cb206c10,0,c7d70000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(cb206c24,0,f19fba60,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(cb206c24,cb206c10,3e9,6a4,c7d6f930,...) at _cv_timedwait_sig+0x252 seltdwait(f19fbc14,f19fbc1c,c849da80,c7d70000,35323a32,...) at seltdwait+0x8a kern_select(c7d70000,2,bfbfebac,0,0,f19fbc5c,20,1,0) at kern_select+0x587 sys_select(c7d70000,f19fbcec,c0f6f44a,c0eedf77,246,...) at sys_select+0x66 syscall(f19fbd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281f24e3, esp = 0xbfbfeb4c, ebp = 0xbfbfeca8 --- Tracing command tail pid 49098 tid 100279 td 0xc871a2e0 sched_switch(c871a2e0,0,104,191,5feeef38,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,68,...) at mi_switch+0x1eb sleepq_switch(c871a2e0,0,c0f24e20,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(3e9,c0a5cc10,c871a2e0,2,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c787da00,68,c0f19974,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c787da00,c787da00,168,c0f19974,3e9,...) at _sleep+0x328 kern_kevent(c871a2e0,4,0,1,f1b4bc44,...) at kern_kevent+0x374 sys_kevent(c871a2e0,f1b4bcec,c0f6f44a,c0eedf77,286,...) at sys_kevent+0x19e syscall(f1b4bd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (363, FreeBSD ELF32, sys_kevent), eip = 0x2816bb7b, esp = 0xbfbfeb1c, ebp = 0xbfbfebd8 --- Tracing command sh pid 49097 tid 100152 td 0xc7fc3b80 Tracing command sshd pid 49096 tid 100095 td 0xc7912b80 sched_switch(c7912b80,0,104,191,5ff24f4f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7912b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c92cf750,0,c0f1da4b,c7912b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c92cf764,0,f183aa60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c92cf764,c92cf750,c0f2743e,6a4,c78b5310,...) at _cv_wait_sig+0x243 seltdwait(c78b5310,58,c7fb2c00,c7912b80,c7912c30,...) at seltdwait+0xa2 kern_select(c7912b80,c,288060b8,288060c0,0,0,20,c790cb10,0) at kern_select+0x587 sys_select(c7912b80,f183acec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f183ad28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 49095 tid 100265 td 0xc870e8a0 sched_switch(c870e8a0,0,104,191,3ddb45b5,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c870e8a0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,cb3ac510,0,c0f1da4b,c870e8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cb3ac524,0,f1b21a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cb3ac524,cb3ac510,c0f2743e,6a4,c8466a48,...) at _cv_wait_sig+0x243 seltdwait(c8466a48,58,c7ebe480,c870e8a0,c870e950,...) at seltdwait+0xa2 kern_select(c870e8a0,a,288060b8,288060c0,0,0,20,c87092c4,0) at kern_select+0x587 sys_select(c870e8a0,f1b21cec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f1b21d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 49094 tid 100269 td 0xc84a8b80 sched_switch(c84a8b80,0,104,191,22cbb70f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c84a8b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,cb7950d0,0,c0f1da4b,c84a8b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cb7950e4,0,f1b2da60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cb7950e4,cb7950d0,c0f2743e,6a4,c7d6fbd0,...) at _cv_wait_sig+0x243 seltdwait(c7d6fbd0,58,c7ebe980,c84a8b80,c19d2a88,...) at seltdwait+0xa2 kern_select(c84a8b80,c,288060b8,288060c0,0,0,20,c8713b10,0) at kern_select+0x587 sys_select(c84a8b80,f1b2dcec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f1b2dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 49062 tid 100195 td 0xc83eeb80 sched_switch(c83eeb80,0,104,191,22a1fa67,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83eeb80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c73987d0,0,c0f1da4b,c83eeb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c73987e4,0,f19eeac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c73987e4,c73987d0,c0f2743e,6a4,f19eeb48,...) at _cv_wait_sig+0x243 seltdwait(ca880b2c,c0f2743e,5a1,c83eeb80,f19eeb48,...) at seltdwait+0xa2 sys_poll(c83eeb80,f19eecec,c0f6f44a,c0d84263,202,...) at sys_poll+0x300 syscall(f19eed28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command sshd pid 49061 tid 100186 td 0xc7d71000 sched_switch(c7d71000,0,104,191,67e43f98,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d71000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c888f450,0,c0f1da4b,c7d71000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c888f464,0,f19cfac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c888f464,c888f450,c0f2743e,6a4,f19cfb48,...) at _cv_wait_sig+0x243 seltdwait(ca880d2c,c0f2743e,5a1,c7d71000,f19cfb48,...) at seltdwait+0xa2 sys_poll(c7d71000,f19cfcec,c,c,202,...) at sys_poll+0x300 syscall(f19cfd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command sshd pid 49060 tid 100235 td 0xc83fe2e0 sched_switch(c83fe2e0,0,104,191,22a202d2,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83fe2e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,cb5e5450,0,c0f1da4b,c83fe2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(cb5e5464,0,f1ac7ac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(cb5e5464,cb5e5450,c0f2743e,6a4,f1ac7b48,...) at _cv_wait_sig+0x243 seltdwait(c7d6902c,c0f2743e,5a1,c83fe2e0,f1ac7b48,...) at seltdwait+0xa2 sys_poll(c83fe2e0,f1ac7cec,c0f6f44a,c0d84263,202,...) at sys_poll+0x300 syscall(f1ac7d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command bash pid 9867 tid 100196 td 0xc83ee8a0 sched_switch(c83ee8a0,0,104,191,15dd7407,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c83ee8a0,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c83eb000,6c,c0f27c03,100,0,...) at sleepq_wait_sig+0x17 _sleep(c83eb000,c83eb088,16c,c0f27c03,0,...) at _sleep+0x35c kern_wait(c83ee8a0,ffffffff,f19f1c60,6,0,...) at kern_wait+0x626 sys_wait4(c83ee8a0,f19f1cec,c0f6f44a,c0f27af2,216,...) at sys_wait4+0x3b syscall(f19f1d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x282c57eb, esp = 0xbfbfe88c, ebp = 0xbfbfe8a8 --- Tracing command sshd pid 9853 tid 100197 td 0xc83ee5c0 sched_switch(c83ee5c0,0,104,191,161e37b7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83ee5c0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7d5e750,0,c0f1da4b,c83ee5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7d5e764,0,f19f4a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7d5e764,c7d5e750,c0f2743e,6a4,c8466e00,...) at _cv_wait_sig+0x243 seltdwait(c8466e00,58,c849d780,c83ee5c0,c83ee670,...) at seltdwait+0xa2 kern_select(c83ee5c0,c,288060b8,288060c0,0,0,20,c820bb10,0) at kern_select+0x587 sys_select(c83ee5c0,f19f4cec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f19f4d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 9576 tid 100185 td 0xc83f0000 sched_switch(c83f0000,0,104,191,58826eb1,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c83f0000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7398810,0,c0f1da4b,c83f0000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7398824,0,f19cbac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7398824,c7398810,c0f2743e,6a4,f19cbb48,...) at _cv_wait_sig+0x243 seltdwait(c7d69a2c,c0f2743e,5a1,c83f0000,f19cbb48,...) at seltdwait+0xa2 sys_poll(c83f0000,f19cbcec,c0f6f44a,c0f10219,202,...) at sys_poll+0x300 syscall(f19cbd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command bash pid 2273 tid 100094 td 0xc7913000 Tracing command csh pid 2269 tid 100121 td 0xc79438a0 Tracing command su pid 2268 tid 100150 td 0xc7bbe8a0 Tracing command bash pid 2263 tid 100171 td 0xc82115c0 Tracing command sshd pid 2262 tid 100173 td 0xc8211000 sched_switch(c8211000,0,104,191,5f8e4889,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c8211000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c73989d0,0,c0f1da4b,c8211000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c73989e4,0,f19a3a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c73989e4,c73989d0,c0f2743e,6a4,c78b6968,...) at _cv_wait_sig+0x243 seltdwait(c78b6968,58,c71a5d00,c8211000,c82110b0,...) at seltdwait+0xa2 kern_select(c8211000,c,288060b8,288060c0,0,0,20,c8207588,0) at kern_select+0x587 sys_select(c8211000,f19a3cec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f19a3d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 2260 tid 100089 td 0xc7576b80 sched_switch(c7576b80,0,104,191,2b277567,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7576b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c8161090,0,c0f1da4b,c7576b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c81610a4,0,f1821ac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c81610a4,c8161090,c0f2743e,6a4,f1821b48,...) at _cv_wait_sig+0x243 seltdwait(c792292c,c0f2743e,5a1,c7576b80,f1821b48,...) at seltdwait+0xa2 sys_poll(c7576b80,f1821cec,c0f6f44a,c0f10219,202,...) at sys_poll+0x300 syscall(f1821d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command getty pid 2251 tid 100128 td 0xc790f2e0 sched_switch(c790f2e0,0,104,191,e47ab7a3,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c790f2e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c790f2e0,c790f2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e9870,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e9870,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e9800,c72e9870,f18edc44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e9800,f18edc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c755cd00,f18edc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7935888,f18edc44,c71a4e80,0,c790f2e0,...) at devfs_read_f+0x7e dofileread(f18edc44,ffffffff,ffffffff,0,f18edc20,...) at dofileread+0x9e kern_readv(c790f2e0,0,f18edc44,f18edc64,1,...) at kern_readv+0x68 sys_read(c790f2e0,f18edcec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f18edd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2250 tid 100162 td 0xc7fc1b80 sched_switch(c7fc1b80,0,104,191,e472934d,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc1b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c7fc1b80,c7fc1b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e9a70,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e9a70,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e9a00,c72e9a70,f1982c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e9a00,f1982c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c755ce00,f1982c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7d48c08,f1982c44,c71a4e80,0,c7fc1b80,...) at devfs_read_f+0x7e dofileread(f1982c44,ffffffff,ffffffff,0,f1982c20,...) at dofileread+0x9e kern_readv(c7fc1b80,0,f1982c44,f1982c64,1,...) at kern_readv+0x68 sys_read(c7fc1b80,f1982cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f1982d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2249 tid 100169 td 0xc8211b80 sched_switch(c8211b80,0,104,191,e47642ae,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c8211b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c8211b80,c8211b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e9c70,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e9c70,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e9c00,c72e9c70,f1997c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e9c00,f1997c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c757f000,f1997c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78b65b0,f1997c44,c71a4e80,0,c8211b80,...) at devfs_read_f+0x7e dofileread(f1997c44,ffffffff,ffffffff,0,f1997c20,...) at dofileread+0x9e kern_readv(c8211b80,0,f1997c44,f1997c64,1,...) at kern_readv+0x68 sys_read(c8211b80,f1997cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f1997d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2248 tid 100119 td 0xc7c6e5c0 sched_switch(c7c6e5c0,0,104,191,e4908004,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7c6e5c0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c7c6e5c0,c7c6e5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e9e70,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e9e70,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e9e00,c72e9e70,f1899c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e9e00,f1899c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c757f100,f1899c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78ccc78,f1899c44,c71a4e80,0,c7c6e5c0,...) at devfs_read_f+0x7e dofileread(f1899c44,ffffffff,ffffffff,0,f1899c20,...) at dofileread+0x9e kern_readv(c7c6e5c0,0,f1899c44,f1899c64,1,...) at kern_readv+0x68 sys_read(c7c6e5c0,f1899cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f1899d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2247 tid 100134 td 0xc7d73000 sched_switch(c7d73000,0,104,191,e487422f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d73000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c7d73000,c7d73000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e070,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e070,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c757e000,c757e070,f18ffc44,1,0,...) at tty_wait+0x71 ttydisc_read(c757e000,f18ffc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c757f200,f18ffc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7d82038,f18ffc44,c71a4e80,0,c7d73000,...) at devfs_read_f+0x7e dofileread(f18ffc44,ffffffff,ffffffff,0,f18ffc20,...) at dofileread+0x9e kern_readv(c7d73000,0,f18ffc44,f18ffc64,1,...) at kern_readv+0x68 sys_read(c7d73000,f18ffcec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f18ffd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2246 tid 100131 td 0xc7d738a0 sched_switch(c7d738a0,0,104,191,e47bf277,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d738a0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c7d738a0,c7d738a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e270,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e270,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c757e200,c757e270,f18f6c44,1,0,...) at tty_wait+0x71 ttydisc_read(c757e200,f18f6c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c757f300,f18f6c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78ccd20,f18f6c44,c71a4e80,0,c7d738a0,...) at devfs_read_f+0x7e dofileread(f18f6c44,ffffffff,ffffffff,0,f18f6c20,...) at dofileread+0x9e kern_readv(c7d738a0,0,f18f6c44,f18f6c64,1,...) at kern_readv+0x68 sys_read(c7d738a0,f18f6cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f18f6d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2245 tid 100164 td 0xc7fc15c0 sched_switch(c7fc15c0,0,104,191,e488f5e0,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc15c0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c7fc15c0,c7fc15c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c757e470,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c757e470,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c757e400,c757e470,f1988c44,1,0,...) at tty_wait+0x71 ttydisc_read(c757e400,f1988c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c757f400,f1988c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c78cc7a8,f1988c44,c71a4e80,0,c7fc15c0,...) at devfs_read_f+0x7e dofileread(f1988c44,ffffffff,ffffffff,0,f1988c20,...) at dofileread+0x9e kern_readv(c7fc15c0,0,f1988c44,f1988c64,1,...) at kern_readv+0x68 sys_read(c7fc15c0,f1988cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f1988d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2244 tid 100102 td 0xc79122e0 sched_switch(c79122e0,0,104,191,e48ef0a7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c79122e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,c79122e0,c79122e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e8670,0,c0f29af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e8670,c10b4570,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e8600,c72e8670,f1859c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e8600,f1859c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c7365300,f1859c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7d6f5b0,f1859c44,c71a4e80,0,c79122e0,...) at devfs_read_f+0x7e dofileread(f1859c44,ffffffff,ffffffff,0,f1859c20,...) at dofileread+0x9e kern_readv(c79122e0,0,f1859c44,f1859c64,1,...) at kern_readv+0x68 sys_read(c79122e0,f1859cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f1859d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 2243 tid 100126 td 0xc790f8a0 sched_switch(c790f8a0,0,104,191,3a517d0b,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c790f8a0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c72e8404,0,c0f1da4b,c790f8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e8470,0,f18e7af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e8470,c72e8404,c0f294c8,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e8400,c72e8470,f18e7c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e8400,f18e7c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c71ad700,f18e7c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c7d484d0,f18e7c44,c71a4e80,0,c790f8a0,...) at devfs_read_f+0x7e dofileread(f18e7c44,ffffffff,ffffffff,0,f18e7c20,...) at dofileread+0x9e kern_readv(c790f8a0,0,f18e7c44,f18e7c64,1,...) at kern_readv+0x68 sys_read(c790f8a0,f18e7cec,c0f6f44a,c0f40be0,246,...) at sys_read+0x4f syscall(f18e7d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28198563, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 2203 tid 100135 td 0xc7d71b80 sched_switch(c7d71b80,0,104,191,d19cbcdf,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d71b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7398a10,0,c0f1da4b,c7d71b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7398a24,0,f1902a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7398a24,c7398a10,c0f2743e,6a4,c78b51f8,...) at _cv_wait_sig+0x243 seltdwait(c78b51f8,58,c71a4e80,c7d71b80,f1902ad0,...) at seltdwait+0xa2 kern_select(c7d71b80,6,bfbfdfa0,0,0,0,20,c7c60b10,4) at kern_select+0x587 sys_select(c7d71b80,f1902cec,c0f6f44a,c0f27c4f,206,...) at sys_select+0x66 syscall(f1902d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281aa4e3, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command bash pid 2162 tid 100176 td 0xc820f5c0 Tracing command csh pid 2158 tid 100174 td 0xc820fb80 Tracing command su pid 2157 tid 100100 td 0xc79448a0 Tracing command bash pid 2152 tid 100112 td 0xc7c70b80 Tracing command sshd pid 2151 tid 100160 td 0xc7fc22e0 sched_switch(c7fc22e0,0,104,191,f05f69d9,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc22e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7398c90,0,c0f1da4b,c7fc22e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7398ca4,0,f197ca60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7398ca4,c7398c90,c0f2743e,6a4,c78b6770,...) at _cv_wait_sig+0x243 seltdwait(c78b6770,58,c7932200,c7fc22e0,c7fc2390,...) at seltdwait+0xa2 kern_select(c7fc22e0,c,288060b8,288060c0,0,0,20,c7fbdb10,0) at kern_select+0x587 sys_select(c7fc22e0,f197ccec,c0f6f44a,c0f27d57,206,...) at sys_select+0x66 syscall(f197cd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfde2c, ebp = 0xbfbfde78 --- Tracing command sshd pid 2149 tid 100116 td 0xc7c70000 sched_switch(c7c70000,0,104,191,f0de1886,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7c70000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c8161250,0,c0f1da4b,c7c70000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8161264,0,f1890ac0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c8161264,c8161250,c0f2743e,6a4,f1890b48,...) at _cv_wait_sig+0x243 seltdwait(c7bce62c,c0f2743e,5a1,c7c70000,f1890b48,...) at seltdwait+0xa2 sys_poll(c7c70000,f1890cec,c0f6f44a,c0f10219,202,...) at sys_poll+0x300 syscall(f1890d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2846566b, esp = 0xbfbfde8c, ebp = 0xbfbfdec8 --- Tracing command cron pid 2110 tid 100151 td 0xc7fc5000 sched_switch(c7fc5000,0,104,191,383749f0,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c7fc5000,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(bf69,c0a5cc10,c7fc5000,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10b5ea4,6c,c0f209f7,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10b5ea4,0,16c,c0f209f7,bf69,...) at _sleep+0x328 kern_nanosleep(c7fc5000,f1961c50,f1961c58,31,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c7fc5000,f1961cec,c0f6f44a,c0f280b0,202,...) at sys_nanosleep+0x6f syscall(f1961d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x28186a47, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sendmail pid 2102 tid 100107 td 0xc75762e0 Tracing command sendmail pid 2098 tid 100155 td 0xc7fc32e0 sched_switch(c7fc32e0,0,104,191,ab270631,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc32e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f196da28,c0a0d57a,c7397e10,0,c7fc32e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c7397e24,0,f196da60,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7397e24,c7397e10,1389,6a4,c78b5038,...) at _cv_timedwait_sig+0x252 seltdwait(f196dc14,f196dc1c,c7ebe700,c7fc32e0,c19ca1d4,...) at seltdwait+0x8a kern_select(c7fc32e0,5,bfbfc510,0,0,f196dc5c,20,5,0) at kern_select+0x587 sys_select(c7fc32e0,f196dcec,c0f6f44a,c0f2800f,206,...) at sys_select+0x66 syscall(f196dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283e54e3, esp = 0xbfbfc47c, ebp = 0xbfbfcfa8 --- Tracing command sshd pid 2090 tid 100161 td 0xc7fc2000 sched_switch(c7fc2000,0,104,191,1af1aeea,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc2000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7397390,0,c0f1da4b,c7fc2000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c73973a4,0,f197fa60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c73973a4,c7397390,c0f2743e,6a4,c78b65e8,...) at _cv_wait_sig+0x243 seltdwait(c78b65e8,58,c7fb2000,c7fc2000,c19d2a88,...) at seltdwait+0xa2 kern_select(c7fc2000,9,2880c0b8,0,0,0,20,c7fbd84c,a) at kern_select+0x587 sys_select(c7fc2000,f197fcec,c0f6f44a,c0f10219,206,...) at sys_select+0x66 syscall(f197fd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284c74e3, esp = 0xbfbfdeec, ebp = 0xbfbfedf8 --- Tracing command moused pid 2030 tid 100159 td 0xc7fc25c0 sched_switch(c7fc25c0,0,104,191,4b91c620,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc25c0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c795b290,0,c0f1da4b,c7fc25c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c795b2a4,0,f1979a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c795b2a4,c795b290,c0f2743e,6a4,c78cc348,...) at _cv_wait_sig+0x243 seltdwait(c78cc348,58,c71a4e80,c7fc25c0,0,...) at seltdwait+0xa2 kern_select(c7fc25c0,400,bfbfea04,0,0,0,20,c7fbf000,1) at kern_select+0x587 sys_select(c7fc25c0,f1979cec,c0f6f44a,c0f40be0,246,...) at sys_select+0x66 syscall(f1979d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281b74e3, esp = 0xbfbfe99c, ebp = 0xbfbfeb28 --- Tracing command ntpd pid 1952 tid 100157 td 0xc7fc2b80 sched_switch(c7fc2b80,0,104,191,7e13c71c,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7fc2b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7398cd0,0,c0f1da4b,c7fc2b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7398ce4,0,f1973a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7398ce4,c7398cd0,c0f2743e,6a4,c7d6f690,...) at _cv_wait_sig+0x243 seltdwait(c7d6f690,58,c71a4e80,c7fc2b80,0,...) at seltdwait+0xa2 kern_select(c7fc2b80,1c,bfbfece8,0,0,0,20,c7fbf588,0) at kern_select+0x587 sys_select(c7fc2b80,f1973cec,c0f6f44a,f1973d80,202,...) at sys_select+0x66 syscall(f1973d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x283664e3, esp = 0xbfbfecbc, ebp = 0xbfbfed88 --- Tracing command nfsd pid 1852 tid 100149 td 0xc7bbeb80 sched_switch(c7bbeb80,0,104,191,2950f7b9,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7bbeb80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f195bbe0,c0a0d57a,c7562e80,0,c7bbeb80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c796ab90,0,f195bc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c796ab90,c7562e80,1388,3b1,0,...) at _cv_timedwait_sig+0x252 svc_run_internal(f195bd14,c09f0528,c7562e80,f195bd28,c0f19db0,...) at svc_run_internal+0x356 svc_thread_start(c7562e80,f195bd28,c0f19db0,3db,c743484c,...) at svc_thread_start+0x10 fork_exit(c0c09820,c7562e80,f195bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1852 tid 100148 td 0xc7c6e000 sched_switch(c7c6e000,0,104,191,b40b669b,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7c6e000,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f1958be0,c0a0d57a,c7562e80,0,c7c6e000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c7891250,0,f1958c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7891250,c7562e80,1388,3b1,0,...) at _cv_timedwait_sig+0x252 svc_run_internal(f1958d14,c09f0528,c7562e80,f1958d28,c0f19db0,...) at svc_run_internal+0x356 svc_thread_start(c7562e80,f1958d28,c0f19db0,3db,c743484c,...) at svc_thread_start+0x10 fork_exit(c0c09820,c7562e80,f1958d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1852 tid 100147 td 0xc7c6e2e0 sched_switch(c7c6e2e0,0,104,191,2761e5a5,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7c6e2e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f1955be0,c0a0d57a,c7562e80,0,c7c6e2e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c796abd0,0,f1955c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c796abd0,c7562e80,1388,3b1,0,...) at _cv_timedwait_sig+0x252 svc_run_internal(f1955d14,c09f0528,c7562e80,f1955d28,c0f19db0,...) at svc_run_internal+0x356 svc_thread_start(c7562e80,f1955d28,c0f19db0,3db,c743484c,...) at svc_thread_start+0x10 fork_exit(c0c09820,c7562e80,f1955d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c12e, eip = 0xc, esp = 0x33, ebp = 0x1 --- Tracing command nfsd pid 1852 tid 100080 td 0xc77f92e0 sched_switch(c77f92e0,0,104,191,29505362,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77f92e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f17fd498,c0a0d57a,c7562e80,0,c77f92e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c7891290,0,f17fd4d0,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7891290,c7562e80,1388,3b1,f17fd518,...) at _cv_timedwait_sig+0x252 svc_run_internal(c77f9490,14,c0f4e3ba,c0f115fb,f17fd810,...) at svc_run_internal+0x356 svc_run(c7562e80,0,c0f115d6,19f,c11feff0,...) at svc_run+0x7b nfsrvd_nfsd(c77f92e0,f17fd810,c,c0a6878c,0,...) at nfsrvd_nfsd+0x1c3 nfssvc_nfsd(c77f92e0,f17fdcec,bfbfe870,c77f92e0,f17fdd1c,...) at nfssvc_nfsd+0x177 sys_nfssvc(c77f92e0,f17fdcec,c0f6f44a,c0f2800f,206,...) at sys_nfssvc+0xeb syscall(f17fdd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280df6db, esp = 0xbfbfe82c, ebp = 0xbfbfea98 --- Tracing command nfsd pid 1845 tid 100133 td 0xc7d732e0 sched_switch(c7d732e0,0,104,191,38a877b7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7d732e0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7ed1350,0,c0f1da4b,c7d732e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7ed1364,0,f18fca60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7ed1364,c7ed1350,c0f2743e,6a4,c7d6f460,...) at _cv_wait_sig+0x243 seltdwait(c7d6f460,58,c71a4e80,c7d732e0,c0f58c27,...) at seltdwait+0xa2 kern_select(c7d732e0,7,bfbfecb4,0,0,0,20,c7d6b2c4,1) at kern_select+0x587 sys_select(c7d732e0,f18fccec,c,c,202,...) at sys_select+0x66 syscall(f18fcd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281864e3, esp = 0xbfbfea9c, ebp = 0xbfbfede8 --- Tracing command mountd pid 1835 tid 100117 td 0xc7c6eb80 sched_switch(c7c6eb80,0,104,191,69e768d6,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7c6eb80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c795b690,0,c0f1da4b,c7c6eb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c795b6a4,0,f1893a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c795b6a4,c795b690,c0f2743e,6a4,c78b5a10,...) at _cv_wait_sig+0x243 seltdwait(c78b5a10,58,c71a4e80,c7c6eb80,c0a6659a,...) at seltdwait+0xa2 kern_select(c7c6eb80,9,bfbfecf4,0,0,0,20,c7c5e000,bfbfecf4) at kern_select+0x587 sys_select(c7c6eb80,f1893cec,c0f6f44a,c0f0f4b7,206,...) at sys_select+0x66 syscall(f1893d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2819c4e3, esp = 0xbfbfec8c, ebp = 0xbfbfedd8 --- Tracing command rpcbind pid 1736 tid 100084 td 0xc7913b80 sched_switch(c7913b80,0,104,191,edef159,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7913b80,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(f180da88,c0a0d57a,c7d5ec90,0,c7913b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c7d5eca4,0,f180dac0,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c7d5eca4,c7d5ec90,7531,6a4,f180db78,...) at _cv_timedwait_sig+0x252 seltdwait(f180dc48,f180dc50,5a1,c7913b80,f180db48,...) at seltdwait+0x8a sys_poll(c7913b80,f180dcec,c0f6f44a,c0f27d40,212,...) at sys_poll+0x300 syscall(f180dd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2814166b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1709 tid 100079 td 0xc77f95c0 sched_switch(c77f95c0,0,104,191,5fed5b99,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77f95c0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7397cd0,0,c0f1da4b,c77f95c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7397ce4,0,f17f9a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7397ce4,c7397cd0,c0f2743e,6a4,c78b6620,...) at _cv_wait_sig+0x243 seltdwait(c78b6620,58,c71a4e80,c77f95c0,1fd,...) at seltdwait+0xa2 kern_select(c77f95c0,9,2847f0ac,0,0,0,20,c7434b10,0) at kern_select+0x587 sys_select(c77f95c0,f17f9cec,c0f6f44a,c0f27af2,246,...) at sys_select+0x66 syscall(f17f9d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2819b4e3, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1502 tid 100083 td 0xc77f88a0 sched_switch(c77f88a0,0,104,191,71efcd1c,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77f88a0,0,c0f24e20,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0d57a,c7398010,0,c0f1da4b,c77f88a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7398024,0,f1809a60,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7398024,c7398010,c0f2743e,6a4,c78cc620,...) at _cv_wait_sig+0x243 seltdwait(c78cc620,58,c71a4e80,c77f88a0,c0f58c27,...) at seltdwait+0xa2 kern_select(c77f88a0,5,bfbfe960,0,0,0,20,c78d7588,34) at kern_select+0x587 sys_select(c77f88a0,f1809cec,c,c,206,...) at sys_select+0x66 syscall(f1809d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x808ba3b, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command softdepflush pid 18 tid 100076 td 0xc74578a0 sched_switch(c74578a0,0,104,191,664967f1,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c74578a0,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1232740,54,c0f553f0,0,0,...) at sleepq_timedwait+0x6b _sleep(c1232740,c1232700,54,c0f553f0,3e8,...) at _sleep+0x342 softdep_flush(0,ed416d28,c0f19db0,3db,c77ef2c4,...) at softdep_flush+0x266 fork_exit(c0c58b40,0,ed416d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed416d60, ebp = 0 --- Tracing command syncer pid 17 tid 100075 td 0xc7457b80 sched_switch(c7457b80,0,104,191,7dc0373a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7457b80,0,c0f24e20,28b,c7457b80,...) at sleepq_switch+0x162 sleepq_timedwait(c122c074,0,ed413c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c122c074,c122c060,3e8,78b,4e20,...) at _cv_timedwait+0x252 sched_sync(0,ed413d28,c0f19db0,3db,c77ef588,...) at sched_sync+0x595 fork_exit(c0ab9d60,0,ed413d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed413d60, ebp = 0 --- Tracing command vnlru pid 16 tid 100074 td 0xc7573000 sched_switch(c7573000,0,104,191,6677e34c,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c7573000,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c77ef84c,60,c0f2fc04,0,0,...) at sleepq_timedwait+0x6b _sleep(c77ef84c,c122c034,260,c0f2fc04,3e8,...) at _sleep+0x342 vnlru_proc(0,ed410d28,c0f19db0,3db,c77ef84c,...) at vnlru_proc+0xe7 fork_exit(c0abbdc0,0,ed410d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed410d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100073 td 0xc75732e0 sched_switch(c75732e0,0,104,191,79910996,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c75732e0,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c122bd08,54,c0f2cb6f,0,0,...) at sleepq_timedwait+0x6b _sleep(c122bd08,c122bd10,54,c0f2cb6f,3e8,...) at _sleep+0x342 buf_daemon(0,ed40dd28,c0f19db0,3db,c77efb10,...) at buf_daemon+0x175 fork_exit(c0aa0270,0,ed40dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40dd60, ebp = 0 --- Tracing command pagezero pid 8 tid 100072 td 0xc75735c0 sched_switch(c75735c0,0,104,191,cf035f97,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75735c0,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12345dc,0,c0f5bf46,0,0,...) at sleepq_timedwait+0x6b _sleep(c12345dc,c1233080,0,c0f5bf46,493e0,...) at _sleep+0x342 vm_pagezero(0,ed40ad28,c0f19db0,3db,c77f0000,...) at vm_pagezero+0xdc fork_exit(c0c9ea00,0,ed40ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40ad60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100071 td 0xc75738a0 sched_switch(c75738a0,0,104,191,101f2473,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c75738a0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12341c8,74,c0f2cb6f,0,0,...) at sleepq_wait+0x63 _sleep(c12341c8,c12341cc,74,c0f2cb6f,0,...) at _sleep+0x372 vm_daemon(0,ed407d28,c0f19db0,3db,c71c72c4,...) at vm_daemon+0x59 fork_exit(c0c98860,0,ed407d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed407d60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100070 td 0xc7573b80 sched_switch(c7573b80,0,104,191,789c4435,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c7573b80,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1234190,54,c0f2cb6f,0,0,...) at sleepq_timedwait+0x6b _sleep(c1234190,c1233080,54,c0f2cb6f,1388,...) at _sleep+0x342 vm_pageout(0,ed404d28,c0f19db0,3db,c71c7588,...) at vm_pageout+0x2c7 fork_exit(c0c99940,0,ed404d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed404d60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100069 td 0xc7575000 sched_switch(c7575000,0,104,191,5f2e33d1,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7575000,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c107ab54,5c,c0d7ea95,0,0,...) at sleepq_wait+0x63 _sleep(c107ab54,c107ab98,5c,c0d7ea95,0,...) at _sleep+0x372 xpt_scanner_thread(0,ed3a1d28,c0f19db0,3db,c71c784c,...) at xpt_scanner_thread+0x47 fork_exit(c048e480,0,ed3a1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3a1d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100068 td 0xc75752e0 sched_switch(c75752e0,0,104,191,a8f5544a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75752e0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c122db78,0,c0f431ff,0,0,...) at sleepq_wait+0x63 _sleep(c122db78,c122db40,0,c0f431ff,0,...) at _sleep+0x372 sctp_iterator_thread(0,ed39ed28,c0f19db0,3db,c71c7b10,...) at sctp_iterator_thread+0x5c fork_exit(c0b4d6f0,0,ed39ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed39ed60, ebp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xc7576000 sched_switch(c7576000,0,104,191,61f5ff6b,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7576000,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c73eea3c,5c,c0f12072,0,0,...) at sleepq_timedwait+0x6b _sleep(c73eea3c,c73eeaf0,5c,c0f12072,3e8,...) at _sleep+0x342 fdc_thread(c73eea00,ed392d28,c0f19db0,3db,c7434000,...) at fdc_thread+0x27d fork_exit(c0d1dcd0,c73eea00,ed392d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed392d60, ebp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xc744b5c0 sched_switch(c744b5c0,0,104,191,5f2e69b9,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c744b5c0,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c747c000,6c,c0f12072,100,0,...) at sleepq_wait_sig+0x17 _sleep(c747c000,c7480488,16c,c0f12072,0,...) at _sleep+0x35c fw_bus_probe_thread(c747c000,ed382d28,c0f19db0,3db,c74342c4,...) at fw_bus_probe_thread+0xa08 fork_exit(c06bdfa0,c747c000,ed382d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed382d60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc74552e0 sched_switch(c74552e0,0,104,191,4cc75aa4,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74552e0,0,c0f24e20,268,c74552e0,...) at sleepq_switch+0x162 sleepq_wait(c745cd34,0,ed36bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cd34,c745cdd4,c0ef86b1,74,c745cd3c,...) at _cv_wait+0x243 usb_process(c745cd2c,ed36bd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c745cd2c,ed36bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed36bd60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc74555c0 sched_switch(c74555c0,0,104,191,56b309a0,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74555c0,0,c0f24e20,268,c74555c0,...) at sleepq_switch+0x162 sleepq_wait(c745cd04,0,ed368cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cd04,c745cdd4,c0ef86b1,74,c745cd0c,...) at _cv_wait+0x243 usb_process(c745ccfc,ed368d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c745ccfc,ed368d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed368d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc74558a0 sched_switch(c74558a0,0,104,191,4c60bff1,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74558a0,0,c0f24e20,268,c74558a0,...) at sleepq_switch+0x162 sleepq_wait(c745ccd4,0,ed365cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745ccd4,c745cdd4,c0ef86b1,74,c745ccdc,...) at _cv_wait+0x243 usb_process(c745cccc,ed365d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c745cccc,ed365d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed365d60, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc7455b80 sched_switch(c7455b80,0,104,191,4c60ac15,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7455b80,0,c0f24e20,268,c7455b80,...) at sleepq_switch+0x162 sleepq_wait(c745cca4,0,ed362cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cca4,c745cdd4,c0ef86b1,74,c745ccac,...) at _cv_wait+0x243 usb_process(c745cc9c,ed362d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c745cc9c,ed362d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed362d60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc74572e0 sched_switch(c74572e0,0,104,191,4c609673,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74572e0,0,c0f24e20,268,c74572e0,...) at sleepq_switch+0x162 sleepq_wait(c7451b5c,0,ed359cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451b5c,c7451bfc,c0ef86b1,74,c7451b64,...) at _cv_wait+0x243 usb_process(c7451b54,ed359d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7451b54,ed359d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed359d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc74575c0 sched_switch(c74575c0,0,104,191,4f2fffac,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74575c0,0,c0f24e20,268,c74575c0,...) at sleepq_switch+0x162 sleepq_wait(c7451b2c,0,ed356cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451b2c,c7451bfc,c0ef86b1,74,c7451b34,...) at _cv_wait+0x243 usb_process(c7451b24,ed356d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7451b24,ed356d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed356d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc7449000 sched_switch(c7449000,0,104,191,4bfc7616,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7449000,0,c0f24e20,268,c7449000,...) at sleepq_switch+0x162 sleepq_wait(c7451afc,0,ed353cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451afc,c7451bfc,c0ef86b1,74,c7451b04,...) at _cv_wait+0x243 usb_process(c7451af4,ed353d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7451af4,ed353d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed353d60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc74492e0 sched_switch(c74492e0,0,104,191,4bfc63f8,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74492e0,0,c0f24e20,268,c74492e0,...) at sleepq_switch+0x162 sleepq_wait(c7451acc,0,ed350cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451acc,c7451bfc,c0ef86b1,74,c7451ad4,...) at _cv_wait+0x243 usb_process(c7451ac4,ed350d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7451ac4,ed350d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed350d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc74495c0 sched_switch(c74495c0,0,104,191,4bfc4e7a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74495c0,0,c0f24e20,268,c74495c0,...) at sleepq_switch+0x162 sleepq_wait(c744db5c,0,ed34ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744db5c,c744dbfc,c0ef86b1,74,c744db64,...) at _cv_wait+0x243 usb_process(c744db54,ed34cd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c744db54,ed34cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34cd60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc74498a0 sched_switch(c74498a0,0,104,191,4c4aaa09,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74498a0,0,c0f24e20,268,c74498a0,...) at sleepq_switch+0x162 sleepq_wait(c744db2c,0,ed349cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744db2c,c744dbfc,c0ef86b1,74,c744db34,...) at _cv_wait+0x243 usb_process(c744db24,ed349d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c744db24,ed349d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed349d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc7449b80 sched_switch(c7449b80,0,104,191,462519bf,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7449b80,0,c0f24e20,268,c7449b80,...) at sleepq_switch+0x162 sleepq_wait(c744dafc,0,ed346cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744dafc,c744dbfc,c0ef86b1,74,c744db04,...) at _cv_wait+0x243 usb_process(c744daf4,ed346d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c744daf4,ed346d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed346d60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc744a000 sched_switch(c744a000,0,104,191,46250852,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a000,0,c0f24e20,268,c744a000,...) at sleepq_switch+0x162 sleepq_wait(c744dacc,0,ed343cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744dacc,c744dbfc,c0ef86b1,74,c744dad4,...) at _cv_wait+0x243 usb_process(c744dac4,ed343d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c744dac4,ed343d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed343d60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc744a2e0 sched_switch(c744a2e0,0,104,191,4624f4ec,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a2e0,0,c0f24e20,268,c744a2e0,...) at sleepq_switch+0x162 sleepq_wait(c7443b5c,0,ed33fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443b5c,c7443bfc,c0ef86b1,74,c7443b64,...) at _cv_wait+0x243 usb_process(c7443b54,ed33fd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7443b54,ed33fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33fd60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc744a5c0 sched_switch(c744a5c0,0,104,191,49654c3d,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a5c0,0,c0f24e20,268,c744a5c0,...) at sleepq_switch+0x162 sleepq_wait(c7443b2c,0,ed33ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443b2c,c7443bfc,c0ef86b1,74,c7443b34,...) at _cv_wait+0x243 usb_process(c7443b24,ed33cd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7443b24,ed33cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33cd60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc744a8a0 sched_switch(c744a8a0,0,104,191,45c0e350,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a8a0,0,c0f24e20,268,c744a8a0,...) at sleepq_switch+0x162 sleepq_wait(c7443afc,0,ed339cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443afc,c7443bfc,c0ef86b1,74,c7443b04,...) at _cv_wait+0x243 usb_process(c7443af4,ed339d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7443af4,ed339d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed339d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc744ab80 sched_switch(c744ab80,0,104,191,45c0d0be,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744ab80,0,c0f24e20,268,c744ab80,...) at sleepq_switch+0x162 sleepq_wait(c7443acc,0,ed336cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443acc,c7443bfc,c0ef86b1,74,c7443ad4,...) at _cv_wait+0x243 usb_process(c7443ac4,ed336d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7443ac4,ed336d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed336d60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc742f000 sched_switch(c742f000,0,104,191,45c0bb2e,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f000,0,c0f24e20,268,c742f000,...) at sleepq_switch+0x162 sleepq_wait(c743bb5c,0,e0ffecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bb5c,c743bbfc,c0ef86b1,74,c743bb64,...) at _cv_wait+0x243 usb_process(c743bb54,e0ffed28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c743bb54,e0ffed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffed60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc742f2e0 sched_switch(c742f2e0,0,104,191,48f70016,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f2e0,0,c0f24e20,268,c742f2e0,...) at sleepq_switch+0x162 sleepq_wait(c743bb2c,0,e0ffbcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bb2c,c743bbfc,c0ef86b1,74,c743bb34,...) at _cv_wait+0x243 usb_process(c743bb24,e0ffbd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c743bb24,e0ffbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffbd60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc742f5c0 sched_switch(c742f5c0,0,104,191,455ce66d,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f5c0,0,c0f24e20,268,c742f5c0,...) at sleepq_switch+0x162 sleepq_wait(c743bafc,0,e0ff8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bafc,c743bbfc,c0ef86b1,74,c743bb04,...) at _cv_wait+0x243 usb_process(c743baf4,e0ff8d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c743baf4,e0ff8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff8d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc742f8a0 sched_switch(c742f8a0,0,104,191,455cd266,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f8a0,0,c0f24e20,268,c742f8a0,...) at sleepq_switch+0x162 sleepq_wait(c743bacc,0,e0ff5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bacc,c743bbfc,c0ef86b1,74,c743bad4,...) at _cv_wait+0x243 usb_process(c743bac4,e0ff5d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c743bac4,e0ff5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff5d60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc7431000 sched_switch(c7431000,0,104,191,455cbad4,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7431000,0,c0f24e20,268,c7431000,...) at sleepq_switch+0x162 sleepq_wait(c7428b5c,0,e0feecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428b5c,c7428bfc,c0ef86b1,74,c7428b64,...) at _cv_wait+0x243 usb_process(c7428b54,e0feed28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7428b54,e0feed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0feed60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc74312e0 sched_switch(c74312e0,0,104,191,41e1f6ec,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74312e0,0,c0f24e20,268,c74312e0,...) at sleepq_switch+0x162 sleepq_wait(c7428b2c,0,e0febcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428b2c,c7428bfc,c0ef86b1,74,c7428b34,...) at _cv_wait+0x243 usb_process(c7428b24,e0febd28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7428b24,e0febd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0febd60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc74315c0 sched_switch(c74315c0,0,104,191,44f8b95f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74315c0,0,c0f24e20,268,c74315c0,...) at sleepq_switch+0x162 sleepq_wait(c7428afc,0,e0fe8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428afc,c7428bfc,c0ef86b1,74,c7428b04,...) at _cv_wait+0x243 usb_process(c7428af4,e0fe8d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7428af4,e0fe8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe8d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc74318a0 sched_switch(c74318a0,0,104,191,44f8a717,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74318a0,0,c0f24e20,268,c74318a0,...) at sleepq_switch+0x162 sleepq_wait(c7428acc,0,e0fe5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428acc,c7428bfc,c0ef86b1,74,c7428ad4,...) at _cv_wait+0x243 usb_process(c7428ac4,e0fe5d28,c0f19db0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c0fe0,c7428ac4,e0fe5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe5d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc72d25c0 sched_switch(c72d25c0,0,104,191,7c08fff4,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d25c0,0,c0f24e20,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10b4ca4,0,c0f12072,2,0,...) at sleepq_timedwait+0x6b _sleep(c10b4ca4,0,0,c0f12072,64,...) at _sleep+0x342 pause(c0f12072,64,c0ed5bfe,111,0,...) at pause+0x47 random_kthread(0,c6f6ad28,c0f19db0,3db,c71c6000,...) at random_kthread+0x1ef fork_exit(c07d2af0,0,c6f6ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6ad60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc72d2b80 sched_switch(c72d2b80,0,104,191,7da605e3,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72d2b80,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b28a8,5c,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c10b28a8,c10b2808,25c,c0f12072,0,...) at _sleep+0x372 g_io_schedule_down(c72d2b80,0,c0f13bd1,6c,c6f64d14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6f64d28,c0f19db0,3db,c71c62c4,...) at g_down_procbody+0x9d fork_exit(c09b2110,0,c6f64d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f64d60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc72d3000 sched_switch(c72d3000,0,104,191,7e4b06ee,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72d3000,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b28a4,5c,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c10b28a4,c10b2828,25c,c0f12072,0,...) at _sleep+0x372 g_io_schedule_up(c72d3000,0,c0f13bd1,5f,c6f61d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6f61d28,c0f19db0,3db,c71c62c4,...) at g_up_procbody+0x9d fork_exit(c09b21b0,0,c6f61d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f61d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc71c92e0 sched_switch(c71c92e0,0,104,191,5ec99ec7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c71c92e0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b289c,5c,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c10b289c,c10b27cc,25c,c0f12072,0,...) at _sleep+0x372 g_run_events(c10c2b00,0,c0f13bd1,79,c6f5ed14,...) at g_run_events+0x54b g_event_procbody(0,c6f5ed28,c0f19db0,3db,c71c62c4,...) at g_event_procbody+0x90 fork_exit(c09b2250,0,c6f5ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5ed60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc75758a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc7575b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100063 td 0xc744b2e0 sched_switch(c744b2e0,0,109,191,3665ec4f,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c744b2e0,...) at mi_switch+0x1eb ithread_loop(c720c6c0,ed385d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c6c0,ed385d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed385d60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc744bb80 sched_switch(c744bb80,0,109,191,3ac9796c,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c744bb80,...) at mi_switch+0x1eb ithread_loop(c720c6d0,ed379d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c6d0,ed379d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed379d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc7455000 sched_switch(c7455000,0,109,191,710b7a80,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c7455000,...) at mi_switch+0x1eb ithread_loop(c720c6e0,ed371d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c6e0,ed371d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed371d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc7457000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc744b000 sched_switch(c744b000,0,109,191,707459d1,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c744b000,...) at mi_switch+0x1eb ithread_loop(c720c700,ed333d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c700,ed333d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed333d60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc742fb80 sched_switch(c742fb80,0,109,191,6a3bb863,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c742fb80,...) at mi_switch+0x1eb ithread_loop(c720c710,e0ff2d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c710,e0ff2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff2d60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc7431b80 sched_switch(c7431b80,0,109,191,cc6bb18d,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c7431b80,...) at mi_switch+0x1eb ithread_loop(c720c720,e0fe2d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c720,e0fe2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe2d60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72d32e0 sched_switch(c72d32e0,0,109,191,7e49bed0,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c72d32e0,...) at mi_switch+0x1eb ithread_loop(c720c730,e0e8cd28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c730,e0e8cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e8cd60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72d35c0 sched_switch(c72d35c0,0,109,191,6aa6795f,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c72d35c0,...) at mi_switch+0x1eb ithread_loop(c720c740,e0e88d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c740,e0e88d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e88d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72d38a0 sched_switch(c72d38a0,0,109,191,3b6bf12d,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c72d38a0,...) at mi_switch+0x1eb ithread_loop(c720c750,c6fb9d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c750,c6fb9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6fb9d60, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc72d3b80 sched_switch(c72d3b80,0,109,191,ffb41a9d,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c72d3b80,...) at mi_switch+0x1eb ithread_loop(c720c760,c6f88d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c760,c6f88d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f88d60, ebp = 0 --- Tracing command intr pid 12 tid 100025 td 0xc73bf2e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xc73bf5c0 sched_switch(c73bf5c0,0,109,191,7e4a59fe,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c73bf5c0,...) at mi_switch+0x1eb ithread_loop(c720c780,c6f7fd28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c780,c6f7fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7fd60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc72d22e0 sched_switch(c72d22e0,0,109,191,7f8fc0d5,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c72d22e0,...) at mi_switch+0x1eb ithread_loop(c720c790,c6f6dd28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c790,c6f6dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6dd60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc71c95c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc71c98a0 sched_switch(c71c98a0,0,109,191,5bd88014,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c71c98a0,...) at mi_switch+0x1eb ithread_loop(c720c7b0,c6f58d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c7b0,c6f58d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f58d60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc71c9b80 sched_switch(c71c9b80,0,109,191,511e01f1,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c71c9b80,...) at mi_switch+0x1eb ithread_loop(c720c7c0,c6f55d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c7c0,c6f55d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f55d60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc720d000 sched_switch(c720d000,0,109,191,7dbff376,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c720d000,...) at mi_switch+0x1eb ithread_loop(c720c7d0,c6f52d28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c7d0,c6f52d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f52d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc720d2e0 sched_switch(c720d2e0,0,109,191,83a6b42c,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c720d2e0,...) at mi_switch+0x1eb ithread_loop(c720c7e0,c6f4fd28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c7e0,c6f4fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4fd60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc720d5c0 sched_switch(c720d5c0,0,109,191,86628f21,...) at sched_switch+0x3bc mi_switch(109,0,c0f1a0d3,554,c720d5c0,...) at mi_switch+0x1eb ithread_loop(c720c7f0,c6f4cd28,c0f19db0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4040,c720c7f0,c6f4cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4cd60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc71c8000 cpustop_handler(3,13,c6f48c50,c0d56916,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,16d8f98,c0d494bb,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f48c5c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d494bb, esp = 0xc6f48c9c, ebp = 0xc6f48ca0 --- spinlock_exit(c10d15e0,c6f48cc4,c0f21562,a0a,c71c8000,...) at spinlock_exit+0x2b sched_idletd(0,c6f48d28,c0f19db0,3db,c71c684c,...) at sched_idletd+0x1e5 fork_exit(c0a43630,0,c6f48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f48d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71c82e0 cpustop_handler(2,13,c6f45c04,c0d56916,c0d5cedc,...) at cpustop_handler+0x28 ipi_nmi_handler(c0d5cedc,c1294140,0,c1261eb8,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f45c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d3b2a5, esp = 0xc6f45c50, ebp = 0xc6f45c50 --- acpi_cpu_c1(c6f45c6c,c107add0,2,301,b8e,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f45ca0,c0d49549,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c3800,2,c6f45ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f45cc4,c0f21562,a0a,c71c82e0,...) at cpu_idle+0x89 sched_idletd(0,c6f45d28,c0f19db0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a43630,0,c6f45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f45d60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71c85c0 sched_switch(c71c85c0,0,608,18c,7e76e583,...) at sched_switch+0x3bc mi_switch(608,0,c0f1fcff,cd,0,...) at mi_switch+0x1eb critical_exit(0,2710,c10c3180,2,c6f42ce8,...) at critical_exit+0xa8 cpu_idle(0,c6f42cc4,c0f21562,a0a,c71c85c0,...) at cpu_idle+0x97 sched_idletd(0,c6f42d28,c0f19db0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a43630,0,c6f42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f42d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71c88a0 cpustop_handler(0,13,c6f3fc04,c0d56916,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,1e30a6c,c0d3b2a5,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f3fc10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d3b2a5, esp = 0xc6f3fc50, ebp = 0xc6f3fc50 --- acpi_cpu_c1(c6f3fc6c,c107add0,0,301,b8e,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f3fca0,c0d49549,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c2b00,2,c6f3fce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f3fcc4,c0f21562,a0a,c71c88a0,...) at cpu_idle+0x89 sched_idletd(0,c6f3fd28,c0f19db0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a43630,0,c6f3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3fd60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc71c8b80 sched_switch(c71c8b80,0,104,191,a3cf5fc5,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c71c8b80,0,c0f24e20,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f24e20,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c71c6b10,6c,c0f27c03,100,0,...) at sleepq_wait_sig+0x17 _sleep(c71c6b10,c71c6b98,16c,c0f27c03,0,...) at _sleep+0x35c kern_wait(c71c8b80,ffffffff,c6f3bc60,0,0,...) at kern_wait+0x626 sys_wait4(c71c8b80,c6f3bcec,c0f6f44a,c0f27a3a,286,...) at sys_wait4+0x3b syscall(c6f3bd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8060df7, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc71c9000 sched_switch(c71c9000,0,104,191,44f69f8f,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c71c9000,0,c0f24e20,268,c71c9000,...) at sleepq_switch+0x162 sleepq_wait(c1231e80,0,c6f38c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c1231e80,c1231e64,c0f4fe37,194,0,...) at _cv_wait+0x243 audit_worker(0,c6f38d28,c0f19db0,3db,c71c7000,...) at audit_worker+0x84 fork_exit(c0c19b70,0,c6f38d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f38d60, ebp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xc77f9b80 sched_switch(c77f9b80,0,104,191,674ae534,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c77f9b80,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c77cfb00,0,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c77cfb00,c77cfb18,0,c0f12072,0,...) at _sleep+0x372 taskqueue_thread_loop(c12941e0,f17f1d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60310,c12941e0,f17f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf17f1d60, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc75755c0 sched_switch(c75755c0,0,104,191,34e83de9,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75755c0,0,c0f24e20,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10b4ca4,0,c0f12072,2,0,...) at sleepq_timedwait+0x6b _sleep(c10b4ca4,0,0,c0f12072,bb8,...) at _sleep+0x342 pause(c0f12072,bb8,124,122,c10b2a08,...) at pause+0x47 deadlkres(0,ed39bd28,c0f19db0,3db,c10b2980,...) at deadlkres+0x324 fork_exit(c09d4260,0,ed39bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed39bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc744b8a0 sched_switch(c744b8a0,0,104,191,7df7cfb8,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744b8a0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c747b140,0,c0f20221,c0f12072,0,...) at sleepq_wait+0x63 msleep_spin(c747b140,c747b158,c0f12072,0,c0f1da4b,...) at msleep_spin+0x21d taskqueue_thread_loop(c748049c,ed37fd28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60310,c748049c,ed37fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc73bf000 sched_switch(c73bf000,0,104,191,674a9cf1,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bf000,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354400,0,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c7354400,c7354418,0,c0f12072,0,...) at _sleep+0x372 taskqueue_thread_loop(c10d186c,c6f85d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60310,c10d186c,c6f85d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f85d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73bf8a0 sched_switch(c73bf8a0,0,104,191,5f323c8a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bf8a0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354480,0,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c7354480,c7354498,0,c0f12072,0,...) at _sleep+0x372 taskqueue_thread_loop(c10b3318,c6f7cd28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60310,c10b3318,c6f7cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc73bfb80 sched_switch(c73bfb80,0,104,191,5f322b82,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bfb80,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20221,c0f12072,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12072,0,c0f1da4b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107da38,c6f79d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60310,c107da38,c6f79d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f79d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc720d8a0 sched_switch(c720d8a0,0,104,191,5f321bce,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c720d8a0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20221,c0f12072,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12072,0,c0f1da4b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107da38,c6f76d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60310,c107da38,c6f76d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f76d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc720db80 sched_switch(c720db80,0,104,191,5f3209a7,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c720db80,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20221,c0f12072,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12072,0,c0f1da4b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107da38,c6f73d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60310,c107da38,c6f73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f73d60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc72d2000 sched_switch(c72d2000,0,104,191,5f2e9045,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d2000,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354500,0,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c7354500,c7354518,0,c0f12072,0,...) at _sleep+0x372 taskqueue_thread_loop(c1232464,c6f70d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60310,c1232464,c6f70d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f70d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc72d28a0 sched_switch(c72d28a0,0,104,191,281b95c4,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d28a0,0,c0f24e20,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7195c40,0,c0f12072,0,0,...) at sleepq_wait+0x63 _sleep(c7195c40,c7195c58,0,c0f12072,0,...) at _sleep+0x372 taskqueue_thread_loop(c10d0280,c6f67d28,c0f19db0,3db,c10b2980,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60310,c10d0280,c6f67d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f67d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc10b2c50 sched_switch(c10b2c50,0,104,191,ff88ff9a,...) at sched_switch+0x3bc mi_switch(104,0,c0f24e20,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c10b2c50,0,c0f24e20,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10b2980,54,c0f21e74,0,0,...) at sleepq_timedwait+0x6b _sleep(c10b2980,0,54,c0f21e74,2710,...) at _sleep+0x342 scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Dumping 208 MB: 193 177 161 145 129 113 97 81 65 49 33 17 1 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump (textdump=0xf1ad672c) at pcpu.h:244 #1 0xc04e2fa9 in db_fncall (dummy1=0xc84445a8, dummy2=0x0, dummy3=0xffffffff, dummy4=0xf1ad67c0 "żgżż") at ../../../ddb/db_command.c:572 #2 0xc04e33df in db_command (last_cmdp=0xc107bbfc, cmd_table=0x0, dopager=0x0) at ../../../ddb/db_command.c:448 #3 0xc04e3494 in db_command_script (command=0xc107cb08 "call doadump") at ../../../ddb/db_command.c:519 #4 0xc04e77d0 in db_script_exec (scriptname=0xc107c460 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at ../../../ddb/db_script.c:302 #5 0xc04e7861 in db_run_cmd (addr=0x1, have_addr=0x0, count=0xc125f1e0, modif=0xf1ad68f8 "") at ../../../ddb/db_script.c:375 #6 0xc04e33a1 in db_command (last_cmdp=0xc107bbfc, cmd_table=0x0, dopager=0x1) at ../../../ddb/db_command.c:448 #7 0xc04e34fa in db_command_loop () at ../../../ddb/db_command.c:501 #8 0xc04e54ed in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:229 #9 0xc0a51d23 in kdb_trap (type=0x3, code=0x0, tf=0xf1ad6aa8) at ../../../kern/subr_kdb.c:625 #10 0xc0d56f4b in trap (frame=0xf1ad6aa8) at ../../../i386/i386/trap.c:722 #11 0xc0d3f8dc in calltrap () at ../../../i386/i386/exception.s:168 #12 0xc0a51b2a in kdb_enter (why=0xc0f1f4c8 "panic", msg=0xc0f1f4c8 "panic") at cpufunc.h:71 #13 0xc0a1d084 in panic (fmt=0xc840729d "null with unref'ed lowervp") at ../../../kern/kern_shutdown.c:600 #14 0xc8405718 in null_checkvp (vp=Variable "vp" is not available. ) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:306 #15 0xc8406e71 in null_lock (ap=0xf1ad6ba8) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:582 #16 0xc0d79a95 in VOP_LOCK1_APV (vop=0xc8408700, a=0xf1ad6ba8) at vnode_if.c:1988 #17 0xc0ac81a8 in _vn_lock (vp=0xc9f5c2b8, flags=0x80100, file=0xc0f2ec9c "../../../kern/vfs_subr.c", line=0x8e7) at vnode_if.h:859 #18 0xc0abc676 in vputx (vp=0xc9f5c2b8, func=0x1) at ../../../kern/vfs_subr.c:2279 #19 0xc0abcf50 in vrele (vp=0xc9f5c2b8) at ../../../kern/vfs_subr.c:2313 #20 0xc09ef408 in exit1 (td=0xc84a7b80, rv=0x0) at ../../../kern/kern_exit.c:367 #21 0xc09ef63d in sys_sys_exit (td=Could not find the frame base for "sys_sys_exit". ) at ../../../kern/kern_exit.c:110 #22 0xc0d56514 in syscall (frame=0xf1ad6d28) at subr_syscall.c:131 #23 0xc0d3f941 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #24 0x00000033 in ?? () (kgdb) f 15 #15 0xc8406e71 in null_lock (ap=0xf1ad6ba8) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_vnops.c:582 582 if (nn != NULL && (lvp = NULLVPTOLOWERVP(vp)) != NULL) { (kgdb) l 577 /* 578 * If we're still active we must ask the lower layer to 579 * lock as ffs has special lock considerations in it's 580 * vop lock. 581 */ 582 if (nn != NULL && (lvp = NULLVPTOLOWERVP(vp)) != NULL) { 583 VI_LOCK_FLAGS(lvp, MTX_DUPOK); 584 VI_UNLOCK(vp); 585 /* 586 * We have to hold the vnode here to solve a potential (kgdb) p *vp $1 = {v_type = VREG, v_tag = 0xc8407309 "null", v_op = 0xc8408700, v_data = 0xc71c3a10, v_mount = 0xc84445a8, v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0xca2ea83c}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0x0, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xc9f5c2e8}, v_cache_dd = 0x0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xc8407309 "null", lo_flags = 0x5730008, lo_data = 0x0, lo_witness = 0xc7169af0}, lk_lock = 0x1, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0x0, pcs = {0x0 }}}, v_interlock = { lock_object = {lo_name = 0xc0f270e9 "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc71621e0}, mtx_lock = 0xc84a7b80}, v_vnlock = 0xc8d401b4, v_holdcnt = 0x1, v_usecount = 0x0, v_iflag = 0x1000, v_vflag = 0x20, v_writecount = 0x0, v_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_mtx = {lock_object = { lo_name = 0xc0f2fbd9 "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xc71679a0}, mtx_lock = 0x4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc9f5c3c4}, bv_root = 0x0, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xc9f5c3d4}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xc10578e0, bo_bsize = 0x4000, bo_object = 0xc8ae02a8, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xc9f5c2b8, __bo_vnode = 0xc9f5c2b8}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) f 14 #14 0xc8405718 in null_checkvp (vp=Variable "vp" is not available. ) at /usr/src/sys/modules/nullfs/../../fs/nullfs/null_subr.c:306 306 panic ("null with unref'ed lowervp"); (kgdb) l 301 int i; u_long *p; 302 printf("vp = %p, unref'ed lowervp\n", (void *)vp); 303 for (p = (u_long *) a, i = 0; i < 8; i++) 304 printf(" %lx", p[i]); 305 printf("\n"); 306 panic ("null with unref'ed lowervp"); 307 } 308 VI_UNLOCK(a->null_lowervp); 309 #ifdef notyet 310 printf("null %x/%d -> %x/%d [%s, %d]\n", (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/vfs_default.c =================================================================== --- /usr/src/sys/kern/vfs_default.c (revision 227502) +++ /usr/src/sys/kern/vfs_default.c (working copy) @@ -844,7 +844,7 @@ out: free(dirbuf, M_TEMP); if (!error) { *buflen = i; - vhold(*dvp); + vref(*dvp); } if (covered) { vput(*dvp); Index: /usr/src/sys/kern/vfs_cache.c =================================================================== --- /usr/src/sys/kern/vfs_cache.c (revision 227502) +++ /usr/src/sys/kern/vfs_cache.c (working copy) @@ -1068,16 +1068,8 @@ vn_vptocnp(struct vnode **vp, struct ucred *cred, CACHE_RLOCK(); error = vn_vptocnp_locked(vp, cred, buf, buflen); - if (error == 0) { - /* - * vn_vptocnp_locked() dropped hold acquired by - * VOP_VPTOCNP immediately after locking the - * cache. Since we are going to drop the cache rlock, - * re-hold the result. - */ - vhold(*vp); + if (error == 0) CACHE_RUNLOCK(); - } return (error); } @@ -1096,6 +1088,9 @@ vn_vptocnp_locked(struct vnode **vp, struct ucred if (ncp != NULL) { if (*buflen < ncp->nc_nlen) { CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT((*vp)->v_mount); + vrele(*vp); + VFS_UNLOCK_GIANT(vfslocked); numfullpathfail4++; error = ENOMEM; SDT_PROBE(vfs, namecache, fullpath, return, error, @@ -1106,18 +1101,23 @@ vn_vptocnp_locked(struct vnode **vp, struct ucred memcpy(buf + *buflen, ncp->nc_name, ncp->nc_nlen); SDT_PROBE(vfs, namecache, fullpath, hit, ncp->nc_dvp, ncp->nc_name, vp, 0, 0); + dvp = *vp; *vp = ncp->nc_dvp; + vref(*vp); + CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(dvp->v_mount); + vrele(dvp); + VFS_UNLOCK_GIANT(vfslocked); + CACHE_RLOCK(); return (0); } SDT_PROBE(vfs, namecache, fullpath, miss, vp, 0, 0, 0, 0); - vhold(*vp); CACHE_RUNLOCK(); vfslocked = VFS_LOCK_GIANT((*vp)->v_mount); vn_lock(*vp, LK_SHARED | LK_RETRY); error = VOP_VPTOCNP(*vp, &dvp, cred, buf, buflen); - VOP_UNLOCK(*vp, 0); - vdrop(*vp); + vput(*vp); VFS_UNLOCK_GIANT(vfslocked); if (error) { numfullpathfail2++; @@ -1128,16 +1128,20 @@ vn_vptocnp_locked(struct vnode **vp, struct ucred *vp = dvp; CACHE_RLOCK(); - if ((*vp)->v_iflag & VI_DOOMED) { + if (dvp->v_iflag & VI_DOOMED) { /* forced unmount */ CACHE_RUNLOCK(); - vdrop(*vp); + vfslocked = VFS_LOCK_GIANT(dvp->v_mount); + vrele(dvp); + VFS_UNLOCK_GIANT(vfslocked); error = ENOENT; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); return (error); } - vdrop(*vp); + /* + * *vp has its use count incremented still. + */ return (0); } @@ -1149,10 +1153,11 @@ static int vn_fullpath1(struct thread *td, struct vnode *vp, struct vnode *rdir, char *buf, char **retbuf, u_int buflen) { - int error, slash_prefixed; + int error, slash_prefixed, vfslocked; #ifdef KDTRACE_HOOKS struct vnode *startvp = vp; #endif + struct vnode *vp1; buflen--; buf[buflen] = '\0'; @@ -1161,6 +1166,7 @@ vn_fullpath1(struct thread *td, struct vnode *vp, SDT_PROBE(vfs, namecache, fullpath, entry, vp, 0, 0, 0, 0); numfullpathcalls++; + vref(vp); CACHE_RLOCK(); if (vp->v_type != VDIR) { error = vn_vptocnp_locked(&vp, td->td_ucred, buf, &buflen); @@ -1168,6 +1174,9 @@ vn_fullpath1(struct thread *td, struct vnode *vp, return (error); if (buflen == 0) { CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); return (ENOMEM); } buf[--buflen] = '/'; @@ -1177,16 +1186,29 @@ vn_fullpath1(struct thread *td, struct vnode *vp, if (vp->v_vflag & VV_ROOT) { if (vp->v_iflag & VI_DOOMED) { /* forced unmount */ CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); error = ENOENT; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); break; } - vp = vp->v_mount->mnt_vnodecovered; + vp1 = vp->v_mount->mnt_vnodecovered; + vref(vp1); + CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); + vp = vp1; + CACHE_RLOCK(); continue; } if (vp->v_type != VDIR) { CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); numfullpathfail1++; error = ENOTDIR; SDT_PROBE(vfs, namecache, fullpath, return, @@ -1198,6 +1220,9 @@ vn_fullpath1(struct thread *td, struct vnode *vp, break; if (buflen == 0) { CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); error = ENOMEM; SDT_PROBE(vfs, namecache, fullpath, return, error, startvp, NULL, 0, 0); @@ -1211,6 +1236,9 @@ vn_fullpath1(struct thread *td, struct vnode *vp, if (!slash_prefixed) { if (buflen == 0) { CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); numfullpathfail4++; SDT_PROBE(vfs, namecache, fullpath, return, ENOMEM, startvp, NULL, 0, 0); @@ -1220,6 +1248,9 @@ vn_fullpath1(struct thread *td, struct vnode *vp, } numfullpathfound++; CACHE_RUNLOCK(); + vfslocked = VFS_LOCK_GIANT(vp->v_mount); + vrele(vp); + VFS_UNLOCK_GIANT(vfslocked); SDT_PROBE(vfs, namecache, fullpath, return, 0, startvp, buf + buflen, 0, 0); Index: /usr/src/sys/fs/pseudofs/pseudofs_vnops.c =================================================================== --- /usr/src/sys/fs/pseudofs/pseudofs_vnops.c (revision 227502) +++ /usr/src/sys/fs/pseudofs/pseudofs_vnops.c (working copy) @@ -410,8 +410,7 @@ pfs_vptocnp(struct vop_vptocnp_args *ap) } *buflen = i; - vhold(*dvp); - vput(*dvp); + VOP_UNLOCK(*dvp, 0); vn_lock(vp, locked | LK_RETRY); vfs_unbusy(mp); @@ -632,8 +631,6 @@ pfs_read(struct vop_read_args *va) (offset = uio->uio_offset) != uio->uio_offset || (resid = uio->uio_resid) != uio->uio_resid || (buflen = offset + resid + 1) < offset || buflen > INT_MAX) { - if (proc != NULL) - PRELE(proc); error = EINVAL; goto ret; } @@ -713,6 +710,13 @@ pfs_iterate(struct thread *td, struct proc *proc, return (0); } +/* Directory entry list */ +struct pfsentry { + STAILQ_ENTRY(pfsentry) link; + struct dirent entry; +}; +STAILQ_HEAD(pfsdirentlist, pfsentry); + /* * Return directory entries. */ @@ -725,12 +729,14 @@ pfs_readdir(struct vop_readdir_args *va) pid_t pid = pvd->pvd_pid; struct proc *p, *proc; struct pfs_node *pn; - struct dirent *entry; struct uio *uio; + struct pfsentry *pfsent, *pfsent2; + struct pfsdirentlist lst; off_t offset; int error, i, resid; - char *buf, *ent; + STAILQ_INIT(&lst); + error = 0; KASSERT(pd->pn_info == vn->v_mount->mnt_data, ("%s(): pn_info does not match mountpoint", __func__)); PFS_TRACE(("%s pid %lu", pd->pn_name, (unsigned long)pid)); @@ -750,8 +756,6 @@ pfs_readdir(struct vop_readdir_args *va) if (resid == 0) PFS_RETURN (0); - /* can't do this while holding the proc lock... */ - buf = malloc(resid, M_IOV, M_WAITOK | M_ZERO); sx_slock(&allproc_lock); pfs_lock(pd); @@ -759,7 +763,6 @@ pfs_readdir(struct vop_readdir_args *va) if (!pfs_visible(curthread, pd, pid, &proc)) { sx_sunlock(&allproc_lock); pfs_unlock(pd); - free(buf, M_IOV); PFS_RETURN (ENOENT); } KASSERT(pid == NO_PID || proc != NULL, @@ -773,57 +776,64 @@ pfs_readdir(struct vop_readdir_args *va) PROC_UNLOCK(proc); pfs_unlock(pd); sx_sunlock(&allproc_lock); - free(buf, M_IOV); PFS_RETURN (0); } } /* fill in entries */ - ent = buf; while (pfs_iterate(curthread, proc, pd, &pn, &p) != -1 && resid >= PFS_DELEN) { - entry = (struct dirent *)ent; - entry->d_reclen = PFS_DELEN; - entry->d_fileno = pn_fileno(pn, pid); + if ((pfsent = malloc(sizeof(struct pfsentry), M_IOV, + M_NOWAIT | M_ZERO)) == NULL) { + error = ENOMEM; + break; + } + pfsent->entry.d_reclen = PFS_DELEN; + pfsent->entry.d_fileno = pn_fileno(pn, pid); /* PFS_DELEN was picked to fit PFS_NAMLEN */ for (i = 0; i < PFS_NAMELEN - 1 && pn->pn_name[i] != '\0'; ++i) - entry->d_name[i] = pn->pn_name[i]; - entry->d_name[i] = 0; - entry->d_namlen = i; + pfsent->entry.d_name[i] = pn->pn_name[i]; + pfsent->entry.d_name[i] = 0; + pfsent->entry.d_namlen = i; switch (pn->pn_type) { case pfstype_procdir: KASSERT(p != NULL, ("reached procdir node with p == NULL")); - entry->d_namlen = snprintf(entry->d_name, + pfsent->entry.d_namlen = snprintf(pfsent->entry.d_name, PFS_NAMELEN, "%d", p->p_pid); /* fall through */ case pfstype_root: case pfstype_dir: case pfstype_this: case pfstype_parent: - entry->d_type = DT_DIR; + pfsent->entry.d_type = DT_DIR; break; case pfstype_file: - entry->d_type = DT_REG; + pfsent->entry.d_type = DT_REG; break; case pfstype_symlink: - entry->d_type = DT_LNK; + pfsent->entry.d_type = DT_LNK; break; default: panic("%s has unexpected node type: %d", pn->pn_name, pn->pn_type); } - PFS_TRACE(("%s", entry->d_name)); + PFS_TRACE(("%s", pfsent->entry.d_name)); + STAILQ_INSERT_TAIL(&lst, pfsent, link); offset += PFS_DELEN; resid -= PFS_DELEN; - ent += PFS_DELEN; } if (proc != NULL) PROC_UNLOCK(proc); pfs_unlock(pd); sx_sunlock(&allproc_lock); - PFS_TRACE(("%zd bytes", ent - buf)); - error = uiomove(buf, ent - buf, uio); - free(buf, M_IOV); + i = 0; + STAILQ_FOREACH_SAFE(pfsent, &lst, link, pfsent2) { + if (error == 0) + error = uiomove(&pfsent->entry, PFS_DELEN, uio); + free(pfsent, M_IOV); + i++; + } + PFS_TRACE(("%zd bytes", i * PFS_DELEN)); PFS_RETURN (error); } Index: /usr/src/sys/fs/nullfs/null_vnops.c =================================================================== --- /usr/src/sys/fs/nullfs/null_vnops.c (revision 227502) +++ /usr/src/sys/fs/nullfs/null_vnops.c (working copy) @@ -808,8 +808,7 @@ null_vptocnp(struct vop_vptocnp_args *ap) #ifdef DIAGNOSTIC NULLVPTOLOWERVP(*dvp); #endif - vhold(*dvp); - vput(*dvp); + VOP_UNLOCK(*dvp, 0); /* keep reference on *dvp */ } else vput(ldvp); Index: /usr/src/sys/fs/devfs/devfs_vnops.c =================================================================== --- /usr/src/sys/fs/devfs/devfs_vnops.c (revision 227502) +++ /usr/src/sys/fs/devfs/devfs_vnops.c (working copy) @@ -261,7 +261,7 @@ devfs_vptocnp(struct vop_vptocnp_args *ap) } else if (vp->v_type == VDIR) { if (dd == dmp->dm_rootdir) { *dvp = vp; - vhold(*dvp); + vref(*dvp); goto finished; } i -= dd->de_dirent->d_namlen; @@ -289,6 +289,8 @@ devfs_vptocnp(struct vop_vptocnp_args *ap) mtx_unlock(&devfs_de_interlock); vholdl(*dvp); VI_UNLOCK(*dvp); + vref(*dvp); + vdrop(*dvp); } else { mtx_unlock(&devfs_de_interlock); error = ENOENT; Index: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c =================================================================== --- /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c (revision 227502) +++ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c (working copy) @@ -1594,7 +1594,7 @@ zfsctl_snapshot_vptocnp(struct vop_vptocnp_args *a *ap->a_buflen -= len; bcopy(sep->se_name, ap->a_buf + *ap->a_buflen, len); mutex_exit(&sdp->sd_lock); - vhold(dvp); + vref(dvp); *ap->a_vpp = dvp; } VN_RELE(dvp); -- Test scenario: nullfs4.sh mount | grep -q /proc || mount -t procfs procfs /proc mount -t nullfs /bin $mntpoint r1=`/bin/ls -l /proc/curproc/file` r2=`$mntpoint/ls -l /proc/curproc/file | sed "s#$mntpoint#/bin#"` if [ "$r1" != "$r2" ]; then echo "/bin/ls -l /proc/curproc/file" echo $r1 echo "$mntpoint/ls -l /proc/curproc/file" echo $r2 fi umount $mntpoint