GENERIC HEAD from 2011-11-16 10:11:55 UTC, r227550M, vmcore.227 KDB: debugger backends: ddb KDB: current backend: ddb 524288K of memory above 4GB ignored Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r227550M: Wed Nov 16 14:52:18 CET 2011 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3531710464 (3368 MB) : Trying to mount root from ufs:/dev/ad4s1a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1a: clean, 130189 free (973 frags, 16152 blocks, 0.1% fragmentation) /dev/ad4s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1f: clean, 137953 free (4425 frags, 16691 blocks, 0.4% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 114670967 free (1239 frags, 14333716 blocks, 0.0% fragmentation) /dev/ad4s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1d: clean, 1397924 free (285804 frags, 139015 blocks, 2.8% fragmentation) /dev/ad4s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s1e: clean, 6563645 free (69477 frags, 811771 blocks, 0.3% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/nss /usr/local/lib/qt4 /usr/local/lib/virtualbox /usr/local/lib/wine a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xe0b20720 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xc7bbf400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f24128,2e687361,38323a63,a34,f18117e8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a570cb,c0f27b6f,c11ff9e0,11c,f1811878,...) at kdb_backtrace+0x2a _witness_debugger(c0f27b6f,c7bbf400,c0f579a6,c71693a0,c0f57638,...) at _witness_debugger+0x25 witness_checkorder(c7bbf400,9,c0f5762f,11c,0,...) at witness_checkorder+0x86f _sx_xlock(c7bbf400,0,c0f5762f,11c,c789915c,...) at _sx_xlock+0x85 ufsdirhash_acquire(e0b206c0,c789915c,f18119f4,e184a9e4,f1811948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c789915c,f18119f4,29e4,f1811934,f1811938,...) at ufsdirhash_add+0x13 ufs_direnter(c7bb7984,c7db4828,f18119f4,f1811b84,e0b20d40,...) at ufs_direnter+0x789 ufs_mkdir(f1811c14,c0f772e8,0,0,f1811bac,...) at ufs_mkdir+0xa36 VOP_MKDIR_APV(c106b2c0,f1811c14,f1811b84,f1811bac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c78a28a0,ffffff9c,28404020,0,1c0,...) at kern_mkdirat+0x291 kern_mkdir(c78a28a0,28404020,0,1c0,f1811d1c,...) at kern_mkdir+0x2e sys_mkdir(c78a28a0,f1811cec,c0f700ea,c0f2872a,202,...) at sys_mkdir+0x29 syscall(f1811d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28174583, esp = 0xbfbfe8cc, ebp = 0xbfbfed78 --- Clearing /tmp (X related). Starting mountd. NFSv4 is disabled Starting nfsd. Recovering vi editor sessions:lock order reversal: 1st 0xc7e71058 ufs (ufs) @ kern/vfs_lookup.c:540 2nd 0xe0b278e0 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:260 3rd 0xc7edb310 ufs (ufs) @ kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f24128,3a632e73,a303632,0,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a570cb,c0f27b88,c11ff9c0,877,f184c344,...) at kdb_backtrace+0x2a _witness_debugger(c0f27b88,c7edb310,c0f168c4,c7169338,c0f2f925,...) at _witness_debugger+0x25 witness_checkorder(c7edb310,9,c0f2f91c,877,0,...) at witness_checkorder+0x86f __lockmgr_args(c7edb310,80100,c7edb37c,0,0,...) at __lockmgr_args+0x834 ffs_lock(f184c468,60,0,80100,c7edb2b8,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c106b2c0,f184c468,c71621e0,c107a340,c7edb2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7edb2b8,80100,c0f2f91c,877,4,...) at _vn_lock+0x78 vget(c7edb2b8,80100,c78a12e0,50,0,...) at vget+0xbb vfs_hash_get(c788cb50,160525,80000,c78a12e0,f184c5ac,...) at vfs_hash_get+0xed ffs_vgetf(c788cb50,160525,80000,f184c5ac,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c7e71000,e0b27880,1,106,0,...) at softdep_sync_buf+0xac9 ffs_syncvnode(c7e71000,1,c11ff9b0,c7e710f8,c7165370,...) at ffs_syncvnode+0x24c ffs_truncate(c7e71000,200,0,880,c71a5300,...) at ffs_truncate+0x817 ufs_direnter(c7e71000,c7edb2b8,f184c918,f184cbc0,0,...) at ufs_direnter+0x971 ufs_makeinode(f184cbc0,c106b840,f184cb04,f184ca60,c0d7b425,...) at ufs_makeinode+0x635 ufs_create(f184cb04,c0f774ae,0,0,f184cb80,...) at ufs_create+0x30 VOP_CREATE_APV(c106b2c0,f184cb04,f184cbc0,f184ca9c,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(f184cb80,f184cc48,1b0,0,c71a5300,...) at vn_open_cred+0x205 vn_open(f184cb80,f184cc48,1b0,c7d291f8,4ec3c2e8,...) at vn_open+0x3b kern_openat(c78a12e0,ffffff9c,28839f60,0,a03,...) at kern_openat+0x1ec kern_open(c78a12e0,28839f60,0,a02,1b0,...) at kern_open+0x35 sys_open(c78a12e0,f184ccec,c0f700ea,c0f2872a,216,...) at sys_open+0x30 syscall(f184cd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x283d65c3, esp = 0xbfbfc4fc, ebp = 0xbfbfc588 --- . Updating motd:. Starting ntpd. Starting default moused. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: backuplock order reversal: 1st 0xc7b691b4 ufs (ufs) @ kern/vfs_mount.c:1209 2nd 0xc7915df0 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1834 KDB: stack backtrace: db_trace_self_wrapper(c0f24128,38313a63,a3433,c0a693dc,8,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a570cb,c0f27b6f,c11ff380,72a,f1947a10,...) at kdb_backtrace+0x2a _witness_debugger(c0f27b6f,c7915df0,c0f0e727,c7169198,c0f53b39,...) at _witness_debugger+0x25 witness_checkorder(c7915df0,9,c0f53b30,72a,c7915e5c,...) at witness_checkorder+0x86f __lockmgr_args(c7915df0,80400,c7915e5c,0,0,...) at __lockmgr_args+0x834 vop_stdlock(f1947b30,4,c0f1e68b,80400,c7915d98,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c1041320,f1947b30,c1233380,c107a340,c7915d98,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7915d98,80400,c0f53b30,72a,c7ba32d4,...) at _vn_lock+0x78 softdep_flushworklist(c7ba32d4,f1947bdc,c81108a0,5dc,0,...) at softdep_flushworklist+0x47 ffs_sync(c7ba32d4,1,f1947c30,4f2,c7025000,...) at ffs_sync+0x2fd dounmount(c7ba32d4,8080000,c81108a0,483,5b85c5ea,...) at dounmount+0x3f8 sys_unmount(c81108a0,f1947cec,c,c,206,...) at sys_unmount+0x331 syscall(f1947d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280debdb, esp = 0xbfbfe64c, ebp = 0xbfbfe718 --- ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 25579 files, 3643014 used, 114670967 free (1239 frags, 14333716 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** ** /dev/label/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 25579 files, 3643014 used, 114670967 free (1239 frags, 14333716 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** fsck -y /tmp watchdogd. Starting inetd. Wed Nov 16 15:08:40 CET 2011 FreeBSD/i386 (x4.osted.lan) (console) login: Nov 16 15:17:03 x4 su: pho to root on /dev/pts/0 tvotohz: negative time difference -63483 sec -999841 usec lock order reversal: 1st 0xc858952c filedesc structure (filedesc structure) @ kern/sys_generic.c:1404 2nd 0xc7db4310 ufs (ufs) @ kern/vfs_vnops.c:924 KDB: stack backtrace: db_trace_self_wrapper(c0f24128,20296572,656b2040,732f6e72,675f7379,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a570cb,c0f27b6f,c11feb40,39c,f19e7978,...) at kdb_backtrace+0x2a _witness_debugger(c0f27b6f,c7db4310,c0f168c4,c7169338,c0f31101,...) at _witness_debugger+0x25 witness_checkorder(c7db4310,9,c0f310f8,39c,c7db437c,...) at witness_checkorder+0x86f __lockmgr_args(c7db4310,80400,c7db437c,0,0,...) at __lockmgr_args+0x834 ffs_lock(f19e7a9c,c77785f0,c77785f0,80400,c7db42b8,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c106b2c0,f19e7a9c,0,c107a340,c7db42b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7db42b8,80400,c0f310f8,39c,0,...) at _vn_lock+0x78 vn_poll(c84a4968,80,c78c1c80,c77f92e0,c81e2200,...) at vn_poll+0x77 sys_poll(c77f92e0,f19e7cec,c0f700ea,c0f28872,282,...) at sys_poll+0x20c syscall(f19e7d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x280feb81, esp = 0xbf5faf60, ebp = 0xbf5fafa8 --- tvotohz: negative time difference -229636091 sec -999998 usec tvotohz: negative time difference -125630459 sec -176707 usec tvotohz: negative time difference -955775860 sec -999205 usec tvotohz: negative time difference -1891695579 sec -919848 usec lock order reversal: 1st 0xc84fd72c filedesc structure (filedesc structure) @ kern/sys_generic.c:1404 2nd 0xc7657df0 devfs (devfs) @ kern/vfs_vnops.c:924 KDB: stack backtrace: db_trace_self_wrapper(c0f24128,2029000a,656b2040,732f6e72,675f7379,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a570cb,c0f27b6f,c11fe8d8,39c,f19b897c,...) at kdb_backtrace+0x2a _witness_debugger(c0f27b6f,c7657df0,c0f0e727,c7169198,c0f31101,...) at _witness_debugger+0x25 witness_checkorder(c7657df0,9,c0f310f8,39c,c7657e5c,...) at witness_checkorder+0x86f __lockmgr_args(c7657df0,80400,c7657e5c,0,0,...) at __lockmgr_args+0x834 vop_stdlock(f19b8a9c,c77785d8,c77785d8,80400,c7657d98,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c1041220,f19b8a9c,0,c107a340,c7657d98,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7657d98,80400,c0f310f8,39c,0,...) at _vn_lock+0x78 vn_poll(c78a7cb0,20,c791b900,c84afb80,c85a3c00,...) at vn_poll+0x77 sys_poll(c84afb80,f19b8cec,c0f700ea,c0f28872,282,...) at sys_poll+0x20c syscall(f19b8d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x280feb81, esp = 0xbf4f9f60, ebp = 0xbf4f9fa8 --- tvotohz: negative time difference -567 sec -930515 usec tvotohz: negative time difference -359585782 sec -999951 usec tvotohz: negative time difference -1981089830 sec -463539 usec tvotohz: negative time difference -83761115 sec -514969 usec tvotohz: negative time difference -63483 sec -999841 usec tvotohz: negative time difference -1946157057 sec -920774 usec : tvotohz: negative time difference -307755003 sec -999998 usec tvotohz: negative time difference -342358011 sec -999998 usec tvotohz: negative time difference -336590843 sec -999998 usec tvotohz: negative time difference -359847926 sec -999951 usec tvotohz: negative time difference -553863793 sec -575199 usec tvotohz: negative time difference -2147483648 sec 0 usec tvotohz: negative time difference -382728187 sec -999998 usec Stopping inetd. Waiting for PIDS: 2183. Shutting down local packages:. Stopping cron. Waiting for PIDS: 2110. Stopping sshd. Waiting for PIDS: 2090. Stopping moused. Waiting for PIDS: 2030, 2030. Stopping ntpd. Waiting for PIDS: 1952. Stopping nfsd. Waiting for PIDS: 1845 1856. Stopping mountd. Waiting for PIDS: 1835. Stopping rpcbind. Waiting for PIDS: 1736. Stopping devd. Waiting for PIDS: 1502. Writing entropy file:. Terminated . Nov 16 21:36:43 x4 syslogd: exiting on signal 15 Nov 16 21:37:03 init: some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # fstat /mnt USER CMD PID FD MOUNT INUM MODE SZ|DV R/W NAME nobody syscall2 16811 wd /mnt 2 drwxrwxrwx 12288 r /mnt # umount -f /mnt # ps -l UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 68102 1 0 20 0 9936 1820 wait Ss u0 0:00.01 -sh (sh) 0 68105 68102 0 20 0 9704 1276 - R+ u0 0:00.00 ps -l # ps -lUnobody UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 65534 16811 1 0 20 0 14556 1372 - T 0- 0:00.00 /tmp/syscall2 # ~KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100003 td 0xc71c88a0 kdb_enter(c0d954fe,c0f240d9,c6f3fba4,c088d3db,c755d000,...) at kdb_enter+0x3a kdb_break(c755d000,1e636e2,c0d3bc65,c755d080,c755d058,...) at kdb_break+0x28 uart_intr(c755d000,0,c12624e0,c6f3fbe8,0,...) at uart_intr+0x11b intr_event_handle(c71c4d00,c6f3fc10,c6f3fc34,0,c73ee200,...) at intr_event_handle+0x76 intr_execute_handlers(c720b0d0,c6f3fc10,0,c6f3fc50,c0d40665,...) at intr_execute_handlers+0x49 lapic_handle_intr(3c,c6f3fc10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0d3bc65, esp = 0xc6f3fc50, ebp = 0xc6f3fc50 --- acpi_cpu_c1(c6f3fc6c,c107ba50,0,301,5c22,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f3fca0,c0d49f09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c3780,2,c6f3fce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f3fcc4,c0f221a2,a0a,c71c88a0,...) at cpu_idle+0x89 sched_idletd(0,c6f3fd28,c0f1a9f0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a443b0,0,c6f3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3fd60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x7aa580 curthread = 0xc71c88a0: pid 11 "idle: cpu0" curpcb = 0xc6f3fd80 fpcurthread = none idlethread = 0xc71c88a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x5eb4580 curthread = 0xc71c85c0: pid 11 "idle: cpu1" curpcb = 0xc6f42d80 fpcurthread = none idlethread = 0xc71c85c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 dynamic pcpu = 0x5eb7580 curthread = 0xc71c82e0: pid 11 "idle: cpu2" curpcb = 0xc6f45d80 fpcurthread = none idlethread = 0xc71c82e0: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 dynamic pcpu = 0x5eba580 curthread = 0xc71c8000: pid 11 "idle: cpu3" curpcb = 0xc6f48d80 fpcurthread = none idlethread = 0xc71c8000: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xc788bb50 /dev/ad4s1a on / (ufs) 0xc788c000 devfs on /dev (devfs) 0xc7ba35a8 /dev/ad4s1f on /home (ufs) 0xc7ba3000 /dev/ad4s1d on /usr (ufs) 0xc788cb50 /dev/ad4s1e on /var (ufs) 0xc788c87c procfs on /proc (procfs) 0xc788c5a8 linprocfs on /usr/compat/linux/proc (linprocfs) 0xc788b87c /dev/label/tmp on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 68102 1 68102 0 Ss+ ttyin 0xc72e8470 sh 18884 0 0 0 DL mdwait 0xc784f000 [md5] 16811 1 96917 65534 T+ (threaded) syscall2 31508466 s syscall2 31508465 s syscall2 31508464 s syscall2 31507951 s syscall2 31507431 s syscall2 31506397 s syscall2 31508456 s syscall2 18 0 0 0 DL sdflush 0xc12333c0 [softdepflush] 17 0 0 0 DL syncer 0xc122ccf4 [syncer] 16 0 0 0 DL vlruwt 0xc77ef84c [vnlru] 9 0 0 0 DL psleep 0xc122c988 [bufdaemon] 8 0 0 0 DL pgzero 0xc123525c [pagezero] 7 0 0 0 DL psleep 0xc1234e48 [vmdaemon] 6 0 0 0 DL psleep 0xc1234e10 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc107b7d4 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc122e7f8 [sctp_iterator] 3 0 0 0 DL - 0xc73eea3c [fdc0] 2 0 0 0 SL - 0xc747c000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xc745cd34 [usbus5] 100057 D - 0xc745cd04 [usbus5] 100056 D - 0xc745ccd4 [usbus5] 100055 D - 0xc745cca4 [usbus5] 100053 D - 0xc7451b5c [usbus4] 100052 D - 0xc7451b2c [usbus4] 100051 D - 0xc7451afc [usbus4] 100050 D - 0xc7451acc [usbus4] 100049 D - 0xc744db5c [usbus3] 100048 D - 0xc744db2c [usbus3] 100047 D - 0xc744dafc [usbus3] 100046 D - 0xc744dacc [usbus3] 100045 D - 0xc7443b5c [usbus2] 100044 D - 0xc7443b2c [usbus2] 100043 D - 0xc7443afc [usbus2] 100042 D - 0xc7443acc [usbus2] 100040 D - 0xc743bb5c [usbus1] 100039 D - 0xc743bb2c [usbus1] 100038 D - 0xc743bafc [usbus1] 100037 D - 0xc743bacc [usbus1] 100035 D - 0xc7428b5c [usbus0] 100034 D - 0xc7428b2c [usbus0] 100033 D - 0xc7428afc [usbus0] 100032 D - 0xc7428acc [usbus0] 14 0 0 0 DL - 0xc10b5924 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xc10b3528 [g_down] 100014 D - 0xc10b3524 [g_up] 100013 D - 0xc10b351c [g_event] 12 0 0 0 WL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100027 I [swi6: Giant taskq] 100025 I [swi5: +] 100024 I [swi2: cambio] 100018 I [swi6: task queue] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc71c6b10 [init] 10 0 0 0 DL audit_wo 0xc1232b00 [audit] 0 0 0 0 DLs (threaded) [kernel] 100077 D - 0xc77f8a40 [mca taskq] 100067 D - 0xc10b5924 [deadlkres] 100061 D - 0xc747b140 [fw0_taskq] 100026 D - 0xc7354400 [thread taskq] 100023 D - 0xc7354480 [kqueue taskq] 100022 D - 0xc73544c0 [acpi_task_2] 100021 D - 0xc73544c0 [acpi_task_1] 100020 D - 0xc73544c0 [acpi_task_0] 100019 D - 0xc7354500 [ffs_trim taskq] 100016 D - 0xc7195c40 [firmware taskq] 100000 D sched 0xc10b3600 [swapper] db:0:ps> allt Tracing command sh pid 68102 tid 113660 td 0xc847a8a0 sched_switch(c847a8a0,0,104,191,48770e99,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c847a8a0,0,c0f25aa0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a0e29a,c72e8404,0,c0f1e68b,c847a8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c72e8470,0,f1a11af0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c72e8470,c72e8404,c0f2a148,527,0,...) at _cv_wait_sig+0x243 tty_wait(c72e8400,c72e8470,f1a11c44,1,0,...) at tty_wait+0x71 ttydisc_read(c72e8400,f1a11c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c71ad700,f1a11c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c843a428,f1a11c44,c71a4e80,0,c847a8a0,...) at devfs_read_f+0x7e dofileread(f1a11c44,ffffffff,ffffffff,0,f1a11c20,...) at dofileread+0x9e kern_readv(c847a8a0,0,f1a11c44,f1a11c64,1,...) at kern_readv+0x68 sys_read(c847a8a0,f1a11cec,c0f700ea,c0f41860,206,...) at sys_read+0x4f syscall(f1a11d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x281fd563, esp = 0xbfbfecac, ebp = 0xbfbfecd8 --- Tracing command md5 pid 18884 tid 32390278 td 0xc84a3b80 sched_switch(c84a3b80,0,104,191,14342a2d,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c84a3b80,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c784f000,5c,c0ebe402,0,0,...) at sleepq_wait+0x63 _sleep(c784f000,c784f020,25c,c0ebe402,0,...) at _sleep+0x372 md_kthread(c784f000,f1b28d28,c0f1a9f0,3db,c7f5d84c,...) at md_kthread+0x11a fork_exit(c0736ce0,c784f000,f1b28d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf1b28d60, ebp = 0 --- Tracing command syscall2 pid 16811 tid 31508466 td 0xc86858a0 sched_switch(c86858a0,0,20a,191,79d386db,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c86858a0,...) at thread_suspend_check+0x2b9 ast(f1b61d28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command syscall2 pid 16811 tid 31508465 td 0xc84b0000 sched_switch(c84b0000,0,20a,191,79d3c9a7,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c84b0000,...) at thread_suspend_check+0x2b9 ast(f1a77d28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command syscall2 pid 16811 tid 31508464 td 0xc84b8000 sched_switch(c84b8000,0,10a,191,269102e0,...) at sched_switch+0x3bc mi_switch(10a,0,c0f2144b,350,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_switch(c84b8000,0,c0f2144b,2a5,c7d2c85c,...) at thread_suspend_switch+0x2a4 thread_single(2,0,c0f1a660,121,c7161340,...) at thread_single+0x2b4 kern_execve(c84b8000,f1a71c3c,0,804c2f9,80553ef,...) at kern_execve+0xb6 sys_fexecve(c84b8000,f1a71cec,c0a4349e,c10c46b8,282,...) at sys_fexecve+0x58 syscall(f1a71d28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (492, FreeBSD ELF32, sys_fexecve), eip = 0x280feb81, esp = 0xbf6fbf60, ebp = 0xbf6fbfa8 --- Tracing command syscall2 pid 16811 tid 31507951 td 0xc84b18a0 sched_switch(c84b18a0,0,20a,191,79d38ddd,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c84b18a0,...) at thread_suspend_check+0x2b9 ast(f1a5fd28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command syscall2 pid 16811 tid 31507431 td 0xc84a18a0 sched_switch(c84a18a0,0,20a,191,79d3afff,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c84a18a0,...) at thread_suspend_check+0x2b9 ast(f19d6d28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command syscall2 pid 16811 tid 31506397 td 0xc847e8a0 sched_switch(c847e8a0,0,20a,191,79d3b38a,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c847e8a0,...) at thread_suspend_check+0x2b9 ast(f1941d28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command syscall2 pid 16811 tid 31508456 td 0xc848d8a0 sched_switch(c848d8a0,0,20a,191,79d391a9,...) at sched_switch+0x3bc mi_switch(20a,0,c0f2144b,330,c7d2c8d4,...) at mi_switch+0x1eb thread_suspend_check(0,0,c0f2643e,f3,c848d8a0,...) at thread_suspend_check+0x2b9 ast(f1b5ed28) at ast+0x431 doreti_ast() at doreti_ast+0x17 Tracing command softdepflush pid 18 tid 100076 td 0xc74578a0 sched_switch(c74578a0,0,104,191,9cd315e5,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c74578a0,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12333c0,54,c0f56070,0,0,...) at sleepq_timedwait+0x6b _sleep(c12333c0,c1233380,54,c0f56070,3e8,...) at _sleep+0x342 softdep_flush(0,ed416d28,c0f1a9f0,3db,c77ef2c4,...) at softdep_flush+0x266 fork_exit(c0c59240,0,ed416d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed416d60, ebp = 0 --- Tracing command syncer pid 17 tid 100075 td 0xc7457b80 sched_switch(c7457b80,0,104,191,9cb78cb9,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7457b80,0,c0f25aa0,28b,c7457b80,...) at sleepq_switch+0x162 sleepq_timedwait(c122ccf4,0,ed413c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c122ccf4,c122cce0,3e8,78b,4e20,...) at _cv_timedwait+0x252 sched_sync(0,ed413d28,c0f1a9f0,3db,c77ef588,...) at sched_sync+0x595 fork_exit(c0aba460,0,ed413d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed413d60, ebp = 0 --- Tracing command vnlru pid 16 tid 100074 td 0xc7573000 sched_switch(c7573000,0,104,191,b88152c7,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c7573000,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c77ef84c,60,c0f30884,0,0,...) at sleepq_timedwait+0x6b _sleep(c77ef84c,c122ccb4,260,c0f30884,3e8,...) at _sleep+0x342 vnlru_proc(0,ed410d28,c0f1a9f0,3db,c77ef84c,...) at vnlru_proc+0xe7 fork_exit(c0abc4c0,0,ed410d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed410d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100073 td 0xc75732e0 sched_switch(c75732e0,0,104,191,b7a59f00,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c75732e0,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c122c988,54,c0f2d7ef,0,0,...) at sleepq_timedwait+0x6b _sleep(c122c988,c122c990,54,c0f2d7ef,3e8,...) at _sleep+0x342 buf_daemon(0,ed40dd28,c0f1a9f0,3db,c77efb10,...) at buf_daemon+0x175 fork_exit(c0aa0ec0,0,ed40dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40dd60, ebp = 0 --- Tracing command pagezero pid 8 tid 100072 td 0xc75735c0 sched_switch(c75735c0,0,104,191,2029e8ae,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75735c0,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c123525c,0,c0f5cbe6,0,0,...) at sleepq_timedwait+0x6b _sleep(c123525c,c1233d00,0,c0f5cbe6,493e0,...) at _sleep+0x342 vm_pagezero(0,ed40ad28,c0f1a9f0,3db,c77f0000,...) at vm_pagezero+0xdc fork_exit(c0c9f3b0,0,ed40ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed40ad60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100071 td 0xc75738a0 sched_switch(c75738a0,0,104,191,87f019a0,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c75738a0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1234e48,74,c0f2d7ef,0,0,...) at sleepq_wait+0x63 _sleep(c1234e48,c1234e4c,74,c0f2d7ef,0,...) at _sleep+0x372 vm_daemon(0,ed407d28,c0f1a9f0,3db,c71c72c4,...) at vm_daemon+0x59 fork_exit(c0c99210,0,ed407d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed407d60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100070 td 0xc7573b80 sched_switch(c7573b80,0,104,191,9325599,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c7573b80,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1234e10,54,c0f2d7ef,0,0,...) at sleepq_timedwait+0x6b _sleep(c1234e10,c1233d00,54,c0f2d7ef,1388,...) at _sleep+0x342 vm_pageout(0,ed404d28,c0f1a9f0,3db,c71c7588,...) at vm_pageout+0x2c7 fork_exit(c0c9a2f0,0,ed404d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed404d60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100069 td 0xc7575000 sched_switch(c7575000,0,104,191,80728585,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7575000,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c107b7d4,5c,c0d7f455,0,0,...) at sleepq_wait+0x63 _sleep(c107b7d4,c107b818,5c,c0d7f455,0,...) at _sleep+0x372 xpt_scanner_thread(0,ed3a1d28,c0f1a9f0,3db,c71c784c,...) at xpt_scanner_thread+0x47 fork_exit(c048e520,0,ed3a1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed3a1d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100068 td 0xc75752e0 sched_switch(c75752e0,0,104,191,be519a87,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75752e0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c122e7f8,0,c0f43e7f,0,0,...) at sleepq_wait+0x63 _sleep(c122e7f8,c122e7c0,0,c0f43e7f,0,...) at _sleep+0x372 sctp_iterator_thread(0,ed39ed28,c0f1a9f0,3db,c71c7b10,...) at sctp_iterator_thread+0x5c fork_exit(c0b4ddf0,0,ed39ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed39ed60, ebp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xc7576000 sched_switch(c7576000,0,104,191,a078cbc5,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c7576000,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c73eea3c,5c,c0f12cd2,0,0,...) at sleepq_timedwait+0x6b _sleep(c73eea3c,c73eeaf0,5c,c0f12cd2,3e8,...) at _sleep+0x342 fdc_thread(c73eea00,ed392d28,c0f1a9f0,3db,c7434000,...) at fdc_thread+0x27d fork_exit(c0d1e680,c73eea00,ed392d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed392d60, ebp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xc744b5c0 sched_switch(c744b5c0,0,104,191,8072c037,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c744b5c0,0,c0f25aa0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f25aa0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c747c000,6c,c0f12cd2,100,0,...) at sleepq_wait_sig+0x17 _sleep(c747c000,c7480488,16c,c0f12cd2,0,...) at _sleep+0x35c fw_bus_probe_thread(c747c000,ed382d28,c0f1a9f0,3db,c74342c4,...) at fw_bus_probe_thread+0xa08 fork_exit(c06be030,c747c000,ed382d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed382d60, ebp = 0 --- Tracing command usb pid 15 tid 100058 td 0xc74552e0 sched_switch(c74552e0,0,104,191,6e24412f,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74552e0,0,c0f25aa0,268,c74552e0,...) at sleepq_switch+0x162 sleepq_wait(c745cd34,0,ed36bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cd34,c745cdd4,c0ef92f1,74,c745cd3c,...) at _cv_wait+0x243 usb_process(c745cd2c,ed36bd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c745cd2c,ed36bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed36bd60, ebp = 0 --- Tracing command usb pid 15 tid 100057 td 0xc74555c0 sched_switch(c74555c0,0,104,191,cb2d4801,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74555c0,0,c0f25aa0,268,c74555c0,...) at sleepq_switch+0x162 sleepq_wait(c745cd04,0,ed368cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cd04,c745cdd4,c0ef92f1,74,c745cd0c,...) at _cv_wait+0x243 usb_process(c745ccfc,ed368d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c745ccfc,ed368d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed368d60, ebp = 0 --- Tracing command usb pid 15 tid 100056 td 0xc74558a0 sched_switch(c74558a0,0,104,191,6dbe5d5a,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74558a0,0,c0f25aa0,268,c74558a0,...) at sleepq_switch+0x162 sleepq_wait(c745ccd4,0,ed365cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745ccd4,c745cdd4,c0ef92f1,74,c745ccdc,...) at _cv_wait+0x243 usb_process(c745cccc,ed365d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c745cccc,ed365d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed365d60, ebp = 0 --- Tracing command usb pid 15 tid 100055 td 0xc7455b80 sched_switch(c7455b80,0,104,191,6dbe4bd4,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7455b80,0,c0f25aa0,268,c7455b80,...) at sleepq_switch+0x162 sleepq_wait(c745cca4,0,ed362cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c745cca4,c745cdd4,c0ef92f1,74,c745ccac,...) at _cv_wait+0x243 usb_process(c745cc9c,ed362d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c745cc9c,ed362d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed362d60, ebp = 0 --- Tracing command usb pid 15 tid 100053 td 0xc74572e0 sched_switch(c74572e0,0,104,191,6dbe3738,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74572e0,0,c0f25aa0,268,c74572e0,...) at sleepq_switch+0x162 sleepq_wait(c7451b5c,0,ed359cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451b5c,c7451bfc,c0ef92f1,74,c7451b64,...) at _cv_wait+0x243 usb_process(c7451b54,ed359d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7451b54,ed359d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed359d60, ebp = 0 --- Tracing command usb pid 15 tid 100052 td 0xc74575c0 sched_switch(c74575c0,0,104,191,c03b5d59,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74575c0,0,c0f25aa0,268,c74575c0,...) at sleepq_switch+0x162 sleepq_wait(c7451b2c,0,ed356cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451b2c,c7451bfc,c0ef92f1,74,c7451b34,...) at _cv_wait+0x243 usb_process(c7451b24,ed356d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7451b24,ed356d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed356d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc7449000 sched_switch(c7449000,0,104,191,6d5a52ee,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7449000,0,c0f25aa0,268,c7449000,...) at sleepq_switch+0x162 sleepq_wait(c7451afc,0,ed353cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451afc,c7451bfc,c0ef92f1,74,c7451b04,...) at _cv_wait+0x243 usb_process(c7451af4,ed353d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7451af4,ed353d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed353d60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc74492e0 sched_switch(c74492e0,0,104,191,6d5a40f9,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74492e0,0,c0f25aa0,268,c74492e0,...) at sleepq_switch+0x162 sleepq_wait(c7451acc,0,ed350cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7451acc,c7451bfc,c0ef92f1,74,c7451ad4,...) at _cv_wait+0x243 usb_process(c7451ac4,ed350d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7451ac4,ed350d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed350d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc74495c0 sched_switch(c74495c0,0,104,191,6d5a2be9,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74495c0,0,c0f25aa0,268,c74495c0,...) at sleepq_switch+0x162 sleepq_wait(c744db5c,0,ed34ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744db5c,c744dbfc,c0ef92f1,74,c744db64,...) at _cv_wait+0x243 usb_process(c744db54,ed34cd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c744db54,ed34cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed34cd60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc74498a0 sched_switch(c74498a0,0,104,191,bccc2f3f,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74498a0,0,c0f25aa0,268,c74498a0,...) at sleepq_switch+0x162 sleepq_wait(c744db2c,0,ed349cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744db2c,c744dbfc,c0ef92f1,74,c744db34,...) at _cv_wait+0x243 usb_process(c744db24,ed349d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c744db24,ed349d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed349d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc7449b80 sched_switch(c7449b80,0,104,191,679b95c8,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7449b80,0,c0f25aa0,268,c7449b80,...) at sleepq_switch+0x162 sleepq_wait(c744dafc,0,ed346cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744dafc,c744dbfc,c0ef92f1,74,c744db04,...) at _cv_wait+0x243 usb_process(c744daf4,ed346d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c744daf4,ed346d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed346d60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc744a000 sched_switch(c744a000,0,104,191,679b8537,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a000,0,c0f25aa0,268,c744a000,...) at sleepq_switch+0x162 sleepq_wait(c744dacc,0,ed343cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c744dacc,c744dbfc,c0ef92f1,74,c744dad4,...) at _cv_wait+0x243 usb_process(c744dac4,ed343d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c744dac4,ed343d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed343d60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc744a2e0 sched_switch(c744a2e0,0,104,191,679b7249,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a2e0,0,c0f25aa0,268,c744a2e0,...) at sleepq_switch+0x162 sleepq_wait(c7443b5c,0,ed33fcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443b5c,c7443bfc,c0ef92f1,74,c7443b64,...) at _cv_wait+0x243 usb_process(c7443b54,ed33fd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7443b54,ed33fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33fd60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc744a5c0 sched_switch(c744a5c0,0,104,191,bb828cbe,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a5c0,0,c0f25aa0,268,c744a5c0,...) at sleepq_switch+0x162 sleepq_wait(c7443b2c,0,ed33ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443b2c,c7443bfc,c0ef92f1,74,c7443b34,...) at _cv_wait+0x243 usb_process(c7443b24,ed33cd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7443b24,ed33cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed33cd60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc744a8a0 sched_switch(c744a8a0,0,104,191,6737dc3b,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744a8a0,0,c0f25aa0,268,c744a8a0,...) at sleepq_switch+0x162 sleepq_wait(c7443afc,0,ed339cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443afc,c7443bfc,c0ef92f1,74,c7443b04,...) at _cv_wait+0x243 usb_process(c7443af4,ed339d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7443af4,ed339d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed339d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc744ab80 sched_switch(c744ab80,0,104,191,6737ca34,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744ab80,0,c0f25aa0,268,c744ab80,...) at sleepq_switch+0x162 sleepq_wait(c7443acc,0,ed336cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7443acc,c7443bfc,c0ef92f1,74,c7443ad4,...) at _cv_wait+0x243 usb_process(c7443ac4,ed336d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7443ac4,ed336d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed336d60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc742f000 sched_switch(c742f000,0,104,191,6737b4e6,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f000,0,c0f25aa0,268,c742f000,...) at sleepq_switch+0x162 sleepq_wait(c743bb5c,0,e0ffecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bb5c,c743bbfc,c0ef92f1,74,c743bb64,...) at _cv_wait+0x243 usb_process(c743bb54,e0ffed28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c743bb54,e0ffed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffed60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc742f2e0 sched_switch(c742f2e0,0,104,191,ba021ddc,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f2e0,0,c0f25aa0,268,c742f2e0,...) at sleepq_switch+0x162 sleepq_wait(c743bb2c,0,e0ffbcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bb2c,c743bbfc,c0ef92f1,74,c743bb34,...) at _cv_wait+0x243 usb_process(c743bb24,e0ffbd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c743bb24,e0ffbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ffbd60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc742f5c0 sched_switch(c742f5c0,0,104,191,66d43b13,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f5c0,0,c0f25aa0,268,c742f5c0,...) at sleepq_switch+0x162 sleepq_wait(c743bafc,0,e0ff8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bafc,c743bbfc,c0ef92f1,74,c743bb04,...) at _cv_wait+0x243 usb_process(c743baf4,e0ff8d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c743baf4,e0ff8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff8d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc742f8a0 sched_switch(c742f8a0,0,104,191,66d42a6f,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c742f8a0,0,c0f25aa0,268,c742f8a0,...) at sleepq_switch+0x162 sleepq_wait(c743bacc,0,e0ff5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c743bacc,c743bbfc,c0ef92f1,74,c743bad4,...) at _cv_wait+0x243 usb_process(c743bac4,e0ff5d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c743bac4,e0ff5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff5d60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc7431000 sched_switch(c7431000,0,104,191,66d41236,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7431000,0,c0f25aa0,268,c7431000,...) at sleepq_switch+0x162 sleepq_wait(c7428b5c,0,e0feecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428b5c,c7428bfc,c0ef92f1,74,c7428b64,...) at _cv_wait+0x243 usb_process(c7428b54,e0feed28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7428b54,e0feed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0feed60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc74312e0 sched_switch(c74312e0,0,104,191,b512c61c,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74312e0,0,c0f25aa0,268,c74312e0,...) at sleepq_switch+0x162 sleepq_wait(c7428b2c,0,e0febcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428b2c,c7428bfc,c0ef92f1,74,c7428b34,...) at _cv_wait+0x243 usb_process(c7428b24,e0febd28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7428b24,e0febd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0febd60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc74315c0 sched_switch(c74315c0,0,104,191,66705d66,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74315c0,0,c0f25aa0,268,c74315c0,...) at sleepq_switch+0x162 sleepq_wait(c7428afc,0,e0fe8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428afc,c7428bfc,c0ef92f1,74,c7428b04,...) at _cv_wait+0x243 usb_process(c7428af4,e0fe8d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7428af4,e0fe8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe8d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc74318a0 sched_switch(c74318a0,0,104,191,66704b03,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c74318a0,0,c0f25aa0,268,c74318a0,...) at sleepq_switch+0x162 sleepq_wait(c7428acc,0,e0fe5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c7428acc,c7428bfc,c0ef92f1,74,c7428ad4,...) at _cv_wait+0x243 usb_process(c7428ac4,e0fe5d28,c0f1a9f0,3db,c7434588,...) at usb_process+0x1a3 fork_exit(c08c1d80,c7428ac4,e0fe5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe5d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xc72d25c0 sched_switch(c72d25c0,0,104,191,ff512601,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d25c0,0,c0f25aa0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10b5924,0,c0f12cd2,2,0,...) at sleepq_timedwait+0x6b _sleep(c10b5924,0,0,c0f12cd2,64,...) at _sleep+0x342 pause(c0f12cd2,64,c0ed679e,111,0,...) at pause+0x47 random_kthread(0,c6f6ad28,c0f1a9f0,3db,c71c6000,...) at random_kthread+0x1ef fork_exit(c07d2d90,0,c6f6ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6ad60, ebp = 0 --- Tracing command geom pid 13 tid 100015 td 0xc72d2b80 sched_switch(c72d2b80,0,104,191,c62861eb,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72d2b80,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b3528,5c,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c10b3528,c10b3488,25c,c0f12cd2,0,...) at _sleep+0x372 g_io_schedule_down(c72d2b80,0,c0f14819,6c,c6f64d14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c6f64d28,c0f1a9f0,3db,c71c62c4,...) at g_down_procbody+0x9d fork_exit(c09b2e50,0,c6f64d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f64d60, ebp = 0 --- Tracing command geom pid 13 tid 100014 td 0xc72d3000 sched_switch(c72d3000,0,104,191,c6302a0f,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c72d3000,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b3524,5c,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c10b3524,c10b34a8,25c,c0f12cd2,0,...) at _sleep+0x372 g_io_schedule_up(c72d3000,0,c0f14819,5f,c6f61d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c6f61d28,c0f1a9f0,3db,c71c62c4,...) at g_up_procbody+0x9d fork_exit(c09b2ef0,0,c6f61d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f61d60, ebp = 0 --- Tracing command geom pid 13 tid 100013 td 0xc71c92e0 sched_switch(c71c92e0,0,104,191,143660bc,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c71c92e0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10b351c,5c,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c10b351c,c10b344c,25c,c0f12cd2,0,...) at _sleep+0x372 g_run_events(c10c3780,0,c0f14819,79,c6f5ed14,...) at g_run_events+0x54b g_event_procbody(0,c6f5ed28,c0f1a9f0,3db,c71c62c4,...) at g_event_procbody+0x90 fork_exit(c09b2f90,0,c6f5ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f5ed60, ebp = 0 --- Tracing command intr pid 12 tid 100066 td 0xc75758a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xc7575b80 sched_switch(c7575b80,0,109,191,666ce64a,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c7575b80,...) at mi_switch+0x1eb ithread_loop(c720c6b0,ed395d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c6b0,ed395d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed395d60, ebp = 0 --- Tracing command intr pid 12 tid 100063 td 0xc744b2e0 sched_switch(c744b2e0,0,109,191,4876e201,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c744b2e0,...) at mi_switch+0x1eb ithread_loop(c720c6c0,ed385d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c6c0,ed385d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed385d60, ebp = 0 --- Tracing command intr pid 12 tid 100060 td 0xc744bb80 sched_switch(c744bb80,0,109,191,5c5002e8,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c744bb80,...) at mi_switch+0x1eb ithread_loop(c720c6d0,ed379d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c6d0,ed379d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed379d60, ebp = 0 --- Tracing command intr pid 12 tid 100059 td 0xc7455000 sched_switch(c7455000,0,109,191,9205b4a4,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c7455000,...) at mi_switch+0x1eb ithread_loop(c720c6e0,ed371d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c6e0,ed371d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed371d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc7457000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xc744b000 sched_switch(c744b000,0,109,191,90e8b860,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c744b000,...) at mi_switch+0x1eb ithread_loop(c720c700,ed333d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c700,ed333d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed333d60, ebp = 0 --- Tracing command intr pid 12 tid 100036 td 0xc742fb80 sched_switch(c742fb80,0,109,191,8ab7b09f,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c742fb80,...) at mi_switch+0x1eb ithread_loop(c720c710,e0ff2d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c710,e0ff2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0ff2d60, ebp = 0 --- Tracing command intr pid 12 tid 100031 td 0xc7431b80 sched_switch(c7431b80,0,109,191,e0928388,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c7431b80,...) at mi_switch+0x1eb ithread_loop(c720c720,e0fe2d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c720,e0fe2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0fe2d60, ebp = 0 --- Tracing command intr pid 12 tid 100030 td 0xc72d32e0 sched_switch(c72d32e0,0,109,191,c62f33ca,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c72d32e0,...) at mi_switch+0x1eb ithread_loop(c720c730,e0e8cd28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c730,e0e8cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e8cd60, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc72d35c0 sched_switch(c72d35c0,0,109,191,29e9daf6,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c72d35c0,...) at mi_switch+0x1eb ithread_loop(c720c740,e0e88d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c740,e0e88d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe0e88d60, ebp = 0 --- Tracing command intr pid 12 tid 100028 td 0xc72d38a0 sched_switch(c72d38a0,0,109,191,5cefb100,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c72d38a0,...) at mi_switch+0x1eb ithread_loop(c720c750,c6fb9d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c750,c6fb9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6fb9d60, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc72d3b80 sched_switch(c72d3b80,0,109,191,f67732cd,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c72d3b80,...) at mi_switch+0x1eb ithread_loop(c720c760,c6f88d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c760,c6f88d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f88d60, ebp = 0 --- Tracing command intr pid 12 tid 100025 td 0xc73bf2e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xc73bf5c0 sched_switch(c73bf5c0,0,109,191,c62fab15,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c73bf5c0,...) at mi_switch+0x1eb ithread_loop(c720c780,c6f7fd28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c780,c6f7fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7fd60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc72d22e0 sched_switch(c72d22e0,0,109,191,94f516a0,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c72d22e0,...) at mi_switch+0x1eb ithread_loop(c720c790,c6f6dd28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c790,c6f6dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f6dd60, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc71c95c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xc71c98a0 sched_switch(c71c98a0,0,109,191,5e0cc4b8,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c71c98a0,...) at mi_switch+0x1eb ithread_loop(c720c7b0,c6f58d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c7b0,c6f58d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f58d60, ebp = 0 --- Tracing command intr pid 12 tid 100010 td 0xc71c9b80 sched_switch(c71c9b80,0,109,191,48ec0799,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c71c9b80,...) at mi_switch+0x1eb ithread_loop(c720c7c0,c6f55d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c7c0,c6f55d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f55d60, ebp = 0 --- Tracing command intr pid 12 tid 100009 td 0xc720d000 sched_switch(c720d000,0,109,191,f496583c,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c720d000,...) at mi_switch+0x1eb ithread_loop(c720c7d0,c6f52d28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c7d0,c6f52d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f52d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc720d2e0 sched_switch(c720d2e0,0,109,191,ff511a1e,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c720d2e0,...) at mi_switch+0x1eb ithread_loop(c720c7e0,c6f4fd28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c7e0,c6f4fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4fd60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc720d5c0 sched_switch(c720d5c0,0,109,191,831d0dae,...) at sched_switch+0x3bc mi_switch(109,0,c0f1ad13,554,c720d5c0,...) at mi_switch+0x1eb ithread_loop(c720c7f0,c6f4cd28,c0f1a9f0,3db,c71c6588,...) at ithread_loop+0x1fb fork_exit(c09f4d60,c720c7f0,c6f4cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f4cd60, ebp = 0 --- Tracing command idle pid 11 tid 100006 td 0xc71c8000 cpustop_handler(3,13,c6f48c04,c0d572d6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,10e3c42,c0d3bc65,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f48c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d3bc65, esp = 0xc6f48c50, ebp = 0xc6f48c50 --- acpi_cpu_c1(c6f48c6c,c107ba50,3,301,5c22,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f48ca0,c0d49f09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c4b00,2,c6f48ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f48cc4,c0f221a2,3c0,c71c8000,...) at cpu_idle+0x89 sched_idletd(0,c6f48d28,c0f1a9f0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a443b0,0,c6f48d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f48d60, ebp = 0 --- Tracing command idle pid 11 tid 100005 td 0xc71c82e0 cpustop_handler(2,13,c6f45c04,c0d572d6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,0,c1262b38,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f45c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d3bc65, esp = 0xc6f45c50, ebp = 0xc6f45c50 --- acpi_cpu_c1(c6f45c6c,c107ba50,2,301,5c22,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f45ca0,c0d49f09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c4480,2,c6f45ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f45cc4,c0f221a2,a0a,c71c82e0,...) at cpu_idle+0x89 sched_idletd(0,c6f45d28,c0f1a9f0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a443b0,0,c6f45d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f45d60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc71c85c0 cpustop_handler(1,13,c6f42c04,c0d572d6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,0,c126280c,c71c684c,...) at ipi_nmi_handler+0x47 trap(c6f42c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d3bc65, esp = 0xc6f42c50, ebp = 0xc6f42c50 --- acpi_cpu_c1(c6f42c6c,c107ba50,1,301,5c22,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f42ca0,c0d49f09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c3e00,2,c6f42ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f42cc4,c0f221a2,a0a,c71c85c0,...) at cpu_idle+0x89 sched_idletd(0,c6f42d28,c0f1a9f0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a443b0,0,c6f42d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f42d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc71c88a0 kdb_enter(c0d954fe,c0f240d9,c6f3fba4,c088d3db,c755d000,...) at kdb_enter+0x3a kdb_break(c755d000,1e636e2,c0d3bc65,c755d080,c755d058,...) at kdb_break+0x28 uart_intr(c755d000,0,c12624e0,c6f3fbe8,0,...) at uart_intr+0x11b intr_event_handle(c71c4d00,c6f3fc10,c6f3fc34,0,c73ee200,...) at intr_event_handle+0x76 intr_execute_handlers(c720b0d0,c6f3fc10,0,c6f3fc50,c0d40665,...) at intr_execute_handlers+0x49 lapic_handle_intr(3c,c6f3fc10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0d3bc65, esp = 0xc6f3fc50, ebp = 0xc6f3fc50 --- acpi_cpu_c1(c6f3fc6c,c107ba50,0,301,5c22,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(0,c6f3fca0,c0d49f09,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c10c3780,2,c6f3fce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c6f3fcc4,c0f221a2,a0a,c71c88a0,...) at cpu_idle+0x89 sched_idletd(0,c6f3fd28,c0f1a9f0,3db,c71c684c,...) at sched_idletd+0x263 fork_exit(c0a443b0,0,c6f3fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f3fd60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc71c8b80 sched_switch(c71c8b80,0,104,191,5b026447,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c71c8b80,0,c0f25aa0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f25aa0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c71c6b10,6c,c0f28883,100,0,...) at sleepq_wait_sig+0x17 _sleep(c71c6b10,c71c6b98,16c,c0f28883,0,...) at _sleep+0x35c kern_wait(c71c8b80,ffffffff,c6f3bc60,2,0,...) at kern_wait+0x626 sys_wait4(c71c8b80,c6f3bcec,c,c,282,...) at sys_wait4+0x3b syscall(c6f3bd28) at syscall+0x2a4 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x8060df7, esp = 0xbfbfe81c, ebp = 0xbfbfe838 --- Tracing command audit pid 10 tid 100001 td 0xc71c9000 sched_switch(c71c9000,0,104,191,666e42fc,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c71c9000,0,c0f25aa0,268,c71c9000,...) at sleepq_switch+0x162 sleepq_wait(c1232b00,0,c6f38c8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c1232b00,c1232ae4,c0f50ab7,194,0,...) at _cv_wait+0x243 audit_worker(0,c6f38d28,c0f1a9f0,3db,c71c7000,...) at audit_worker+0x84 fork_exit(c0c1a270,0,c6f38d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f38d60, ebp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xc7800b80 sched_switch(c7800b80,0,104,191,3425db18,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c7800b80,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c77f8a40,0,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c77f8a40,c77f8a58,0,c0f12cd2,0,...) at _sleep+0x372 taskqueue_thread_loop(c1294e60,ed419d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60f60,c1294e60,ed419d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed419d60, ebp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xc75755c0 sched_switch(c75755c0,0,104,191,315255e1,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c75755c0,0,c0f25aa0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c10b5924,0,c0f12cd2,2,0,...) at sleepq_timedwait+0x6b _sleep(c10b5924,0,0,c0f12cd2,bb8,...) at _sleep+0x342 pause(c0f12cd2,bb8,124,122,c10b3688,...) at pause+0x47 deadlkres(0,ed39bd28,c0f1a9f0,3db,c10b3600,...) at deadlkres+0x324 fork_exit(c09d4f80,0,ed39bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed39bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xc744b8a0 sched_switch(c744b8a0,0,104,191,fa6202f7,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c744b8a0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c747b140,0,c0f20e61,c0f12cd2,0,...) at sleepq_wait+0x63 msleep_spin(c747b140,c747b158,c0f12cd2,0,c0f1e68b,...) at msleep_spin+0x21d taskqueue_thread_loop(c748049c,ed37fd28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60f60,c748049c,ed37fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed37fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc73bf000 sched_switch(c73bf000,0,104,191,87f3d400,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bf000,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354400,0,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c7354400,c7354418,0,c0f12cd2,0,...) at _sleep+0x372 taskqueue_thread_loop(c10d24ec,c6f85d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60f60,c10d24ec,c6f85d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f85d60, ebp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xc73bf8a0 sched_switch(c73bf8a0,0,104,191,80768b60,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bf8a0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354480,0,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c7354480,c7354498,0,c0f12cd2,0,...) at _sleep+0x372 taskqueue_thread_loop(c10b3f98,c6f7cd28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60f60,c10b3f98,c6f7cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f7cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc73bfb80 sched_switch(c73bfb80,0,104,191,80767966,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c73bfb80,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20e61,c0f12cd2,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12cd2,0,c0f1e68b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107e6b8,c6f79d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60f60,c107e6b8,c6f79d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f79d60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc720d8a0 sched_switch(c720d8a0,0,104,191,80766ad1,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c720d8a0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20e61,c0f12cd2,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12cd2,0,c0f1e68b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107e6b8,c6f76d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60f60,c107e6b8,c6f76d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f76d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc720db80 sched_switch(c720db80,0,104,191,8076591b,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c720db80,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c73544c0,0,c0f20e61,c0f12cd2,0,...) at sleepq_wait+0x63 msleep_spin(c73544c0,c73544d8,c0f12cd2,0,c0f1e68b,...) at msleep_spin+0x21d taskqueue_thread_loop(c107e6b8,c6f73d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0x8f fork_exit(c0a60f60,c107e6b8,c6f73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f73d60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc72d2000 sched_switch(c72d2000,0,104,191,8072e595,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d2000,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7354500,0,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c7354500,c7354518,0,c0f12cd2,0,...) at _sleep+0x372 taskqueue_thread_loop(c12330e4,c6f70d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60f60,c12330e4,c6f70d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f70d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc72d28a0 sched_switch(c72d28a0,0,104,191,4937af7e,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c72d28a0,0,c0f25aa0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c7195c40,0,c0f12cd2,0,0,...) at sleepq_wait+0x63 _sleep(c7195c40,c7195c58,0,c0f12cd2,0,...) at _sleep+0x372 taskqueue_thread_loop(c10d0f00,c6f67d28,c0f1a9f0,3db,c10b3600,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a60f60,c10d0f00,c6f67d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc6f67d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc10b38d0 sched_switch(c10b38d0,0,104,191,c339e3e,...) at sched_switch+0x3bc mi_switch(104,0,c0f25aa0,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c10b38d0,0,c0f25aa0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10b3600,54,c0f22ab4,0,0,...) at sleepq_timedwait+0x6b _sleep(c10b3600,0,54,c0f22ab4,2710,...) at _sleep+0x342 scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 3435 MB Dumping 175 MB: 160 144 128 112 96 80 64 48 32 16 Dump complete = 0 db:0:doadump> reset (kgdb) thread 83 [Switching to thread 83 (Thread 31508464)]#0 sched_switch (td=0xc84b8000, newtd=0xc71c8000, flags=0x10a) at ../../../kern/sched_ule.c:1859 1859 cpuid = PCPU_GET(cpuid); (kgdb) bt #0 sched_switch (td=0xc84b8000, newtd=0xc71c8000, flags=0x10a) at ../../../kern/sched_ule.c:1859 #1 0xc0a2702b in mi_switch (flags=0x10a, newtd=0x0) at ../../../kern/kern_synch.c:448 #2 0xc0a2d474 in thread_suspend_switch (td=0xc84b8000) at ../../../kern/kern_thread.c:850 #3 0xc0a2d7d4 in thread_single (mode=0x2) at ../../../kern/kern_thread.c:694 #4 0xc09ecff6 in kern_execve (td=0xc84b8000, args=0xf1a71c3c, mac_p=0x0) at ../../../kern/kern_exec.c:290 #5 0xc09ee448 in sys_fexecve (td=0xc84b8000, uap=0xf1a71cec) at ../../../kern/kern_exec.c:230 #6 0xc0d56ed4 in syscall (frame=0xf1a71d28) at subr_syscall.c:131 #7 0xc0d40301 in Xint0x80_syscall () at ../../../i386/i386/exception.s:266 #8 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 5 #5 0xc09ee448 in sys_fexecve (td=0xc84b8000, uap=0xf1a71cec) at ../../../kern/kern_exec.c:230 230 error = kern_execve(td, &args, NULL); (kgdb) p *uap $1 = {fd_l_ = 0xf1a71cec "¿", fd = 0xf5, fd_r_ = 0xf1a71cf0 "¿\004\b¿S\005\b¿Q\v¿", argv_l_ = 0xf1a71cf0 "¿\004\b¿S\005\b¿Q\v¿", argv = 0x804c2f9, argv_r_ = 0xf1a71cf4 "¿S\005\b¿Q\v¿", envv_l_ = 0xf1a71cf4 "¿S\005\b¿Q\v¿", envv = 0x80553ef, envv_r_ = 0xf1a71cf8 "¿Q\v¿"} (kgdb) f 4 #4 0xc09ecff6 in kern_execve (td=0xc84b8000, args=0xf1a71c3c, mac_p=0x0) at ../../../kern/kern_exec.c:290 290 if (thread_single(SINGLE_BOUNDARY)) { (kgdb) l 285 args->begin_envv - args->begin_argv); 286 AUDIT_ARG_ENVV(args->begin_envv, args->envc, 287 args->endp - args->begin_envv); 288 if (p->p_flag & P_HADTHREADS) { 289 PROC_LOCK(p); 290 if (thread_single(SINGLE_BOUNDARY)) { 291 PROC_UNLOCK(p); 292 exec_free_args(args); 293 return (ERESTART); /* Try again later. */ 294 } (kgdb) p *p $2 = {p_list = {le_next = 0xc77ef2c4, le_prev = 0xc7f5d84c}, p_threads = {tqh_first = 0xc86858a0, tqh_last = 0xc848d8a8}, p_slock = {lock_object = {lo_name = 0xc0f1e972 "process slock", lo_flags = 0xb0000, lo_data = 0x0, lo_witness = 0x0}, mtx_lock = 0x4}, p_ucred = 0xc892c300, p_fd = 0xc78a4900, p_fdtol = 0x0, p_stats = 0xc7d2b100, p_limit = 0xc71b0400, p_limco = { c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0x0, c_arg = 0x0, c_func = 0, c_lock = 0xc7d2c8d4, c_flags = 0x0, c_cpu = 0x0}, p_sigacts = 0xc8875000, p_flag = 0x10490182, p_state = PRS_NORMAL, p_pid = 0x41ab, p_hash = {le_next = 0x0, le_prev = 0xc719b6ac}, p_pglist = {le_next = 0x0, le_prev = 0xc717e648}, p_pptr = 0xc71c6b10, p_sibling = { le_next = 0x0, le_prev = 0xc7f5c604}, p_children = {lh_first = 0x0}, p_mtx = {lock_object = {lo_name = 0xc0f1e965 "process lock", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xc7161340}, mtx_lock = 0x4}, p_ksi = 0xc773b460, p_sigqueue = {sq_signals = {__bits = {0x6101, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x100, 0x0, 0x0, 0x0}}, sq_list = { tqh_first = 0xc7738c30, tqh_last = 0xc7733eb0}, sq_proc = 0xc7d2c84c, sq_flags = 0x1}, p_oppid = 0x0, p_dbg_child = 0x0, p_vmspace = 0xc84d9658, p_swtick = 0x139bb64, p_realtimer = { it_interval = {tv_sec = 0x0, tv_usec = 0x0}, it_value = {tv_sec = 0x0, tv_usec = 0x0}}, p_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0, ru_isrss = 0x0, ru_minflt = 0x0, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x0, ru_nivcsw = 0x0}, p_rux = {rux_runtime = 0x2f6a4c, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x6be, rux_tu = 0x6be}, p_crux = {rux_runtime = 0x0, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0x0, rux_tu = 0x0}, p_profthreads = 0x0, p_exitthreads = 0x0, p_traceflag = 0x0, p_tracevp = 0x0, p_tracecred = 0x0, p_textvp = 0xc8b586cc, p_lock = 0x0, p_sigiolst = {slh_first = 0x0}, p_sigparent = 0x14, p_sig = 0x0, p_code = 0x0, p_stops = 0x0, p_stype = 0x0, p_step = 0x0, p_pfsflags = 0x0, p_nlminfo = 0x0, p_aioinfo = 0x0, p_singlethread = 0xc84b8000, p_suspcount = 0x7, p_xthread = 0x0, p_boundary_count = 0x4, p_pendingcnt = 0x3, p_itimers = 0x0, p_procdesc = 0x0, p_magic = 0xbeefface, p_osrel = 0xf4240, p_comm = "syscall2", '\0' , p_pgrp = 0xc717e640, p_sysent = 0xc1074900, p_args = 0xc791d3c0, p_cpulimit = 0x7fffffffffffffff, p_nice = 0x0, p_fibnum = 0x0, p_xstat = 0x13, p_klist = {kl_list = {slh_first = 0x0}, kl_lock = 0xc09e83e0 , kl_unlock = 0xc09e8410 , kl_assert_locked = 0xc09e8760 , kl_assert_unlocked = 0xc09e8730 , kl_lockarg = 0xc7d2c8d4}, p_numthreads = 0x7, p_md = {md_ldt = 0x0}, p_itcallout = {c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0xe0ab03e8}}, c_time = 0x139bf4d, c_arg = 0xc7d2c84c, c_func = 0xc0a30670 , c_lock = 0x0, c_flags = 0x12, c_cpu = 0x0}, p_acflag = 0x1, p_peers = 0x0, p_leader = 0xc7d2c84c, p_emuldata = 0x0, p_label = 0x0, p_sched = 0xc7d2cb10, p_ktr = {stqh_first = 0x0, stqh_last = 0xc7d2cae4}, p_mqnotifier = {lh_first = 0x0}, p_dtrace = 0x0, p_pwait = {cv_description = 0xc0f1f121 "ppwait", cv_waiters = 0x0}, p_dbgwait = {cv_description = 0xc0f1f128 "dbgwait", cv_waiters = 0x0}, p_prev_runtime = 0x0, p_racct = 0x0} (kgdb) f 3 #3 0xc0a2d7d4 in thread_single (mode=0x2) at ../../../kern/kern_thread.c:694 694 thread_suspend_switch(td); (kgdb) l 689 stopme: 690 /* 691 * Wake us up when everyone else has suspended. 692 * In the mean time we suspend as well. 693 */ 694 thread_suspend_switch(td); 695 remaining = calc_remaining(p, mode); 696 } 697 if (mode == SINGLE_EXIT) { 698 /* (kgdb) info loc td = (struct thread *) 0xc84b8000 td2 = (struct thread *) 0x0 p = (struct proc *) 0xc7d2c84c remaining = Variable "remaining" is not available. (kgdb) p *td $3 = {td_lock = 0xc10c4b00, td_proc = 0xc7d2c84c, td_plist = {tqe_next = 0xc84b18a0, tqe_prev = 0xc84b0008}, td_runq = {tqe_next = 0x0, tqe_prev = 0xc10c4eb8}, td_slpq = {tqe_next = 0x0, tqe_prev = 0xc868b2c0}, td_lockq = {tqe_next = 0xc8b108a0, tqe_prev = 0xf19c7b94}, td_hash = {le_next = 0x0, le_prev = 0xc719ffc0}, td_cpuset = 0xc71a2e38, td_sel = 0xc8707880, td_sleepqueue = 0xc868b2c0, td_turnstile = 0xc84b9980, td_umtxq = 0xc7b8f700, td_tid = 0x1e0c7f0, td_sigqueue = {sq_signals = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_kill = {__bits = {0x0, 0x0, 0x0, 0x0}}, sq_list = {tqh_first = 0x0, tqh_last = 0xc84b8068}, sq_proc = 0xc7d2c84c, sq_flags = 0x1}, td_lend_user_pri = 0xff, td_flags = 0x44, td_inhibitors = 0x1, td_pflags = 0x0, td_dupfd = 0x0, td_sqqueue = 0x0, td_wchan = 0x0, td_wmesg = 0x0, td_lastcpu = 0x3, td_oncpu = 0xff, td_owepreempt = 0x0, td_tsqueue = 0x0, td_locks = 0x0, td_rw_rlocks = 0x0, td_lk_slocks = 0x0, td_blocked = 0x0, td_lockname = 0x0, td_contested = {lh_first = 0x0}, td_sleeplocks = 0xc11feb40, td_intr_nesting_level = 0x0, td_pinned = 0x0, td_ucred = 0xc892c300, td_estcpu = 0x0, td_slptick = 0x16648a2, td_blktick = 0x0, td_swvoltick = 0x16648a2, td_ru = {ru_utime = {tv_sec = 0x0, tv_usec = 0x0}, ru_stime = {tv_sec = 0x0, tv_usec = 0x0}, ru_maxrss = 0x0, ru_ixrss = 0x0, ru_idrss = 0x0, ru_isrss = 0x0, ru_minflt = 0x1, ru_majflt = 0x0, ru_nswap = 0x0, ru_inblock = 0x0, ru_oublock = 0x0, ru_msgsnd = 0x0, ru_msgrcv = 0x0, ru_nsignals = 0x0, ru_nvcsw = 0x13, ru_nivcsw = 0x2}, td_rux = {rux_runtime = 0x4f1b5, rux_uticks = 0x0, rux_sticks = 0x0, rux_iticks = 0x0, rux_uu = 0x0, rux_su = 0xad, rux_tu = 0xad}, td_incruntime = 0x0, td_runtime = 0x4f1b5, td_pticks = 0x0, td_sticks = 0x0, td_iticks = 0x0, td_uticks = 0x0, td_intrval = 0x0, td_oldsigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_sigmask = {__bits = {0x0, 0x0, 0x0, 0x0}}, td_generation = 0x15, td_sigstk = {ss_sp = 0x0, ss_size = 0x0, ss_flags = 0x0}, td_xsig = 0x0, td_profil_addr = 0x0, td_profil_ticks = 0x0, td_name = "syscall2", '\0' , td_fpop = 0x0, td_dbgflags = 0x0, td_dbgksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 0x0, si_errno = 0x0, si_code = 0x0, si_pid = 0x0, si_uid = 0x0, si_status = 0x0, si_addr = 0x0, si_value = {sival_int = 0x0, sival_ptr = 0x0, sigval_int = 0x0, sigval_ptr = 0x0}, _reason = {_fault = { _trapno = 0x0}, _timer = {_timerid = 0x0, _overrun = 0x0}, _mesgq = {_mqd = 0x0}, _poll = {_band = 0x0}, __spare__ = {__spare1__ = 0x0, __spare2__ = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}, ksi_flags = 0x0, ksi_sigq = 0x0}, td_ng_outbound = 0x0, td_osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, td_map_def_user = 0x0, td_dbg_forked = 0x0, td_rqindex = 0x2f, td_base_pri = 0x98, td_priority = 0x98, td_pri_class = 0x3, td_user_pri = 0x78, td_base_user_pri = 0x78, td_pcb = 0xf1a71d80, td_state = TDS_INHIBITED, td_retval = {0x0, 0x1}, td_slpcallout = {c_links = {sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0x0, c_arg = 0x0, c_func = 0, c_lock = 0x0, c_flags = 0x10, c_cpu = 0x0}, td_frame = 0xf1a71d28, td_kstack_obj = 0xc81ff118, td_kstack = 0xf1a70000, td_kstack_pages = 0x2, td_critnest = 0x1, td_md = { md_spinlock_count = 0x1, md_saved_flags = 0x246}, td_sched = 0xc84b82b0, td_ar = 0x0, td_lprof = {{lh_first = 0x0}, {lh_first = 0x0}}, td_dtrace = 0x0, td_errno = 0xffffffff, td_vnet = 0x0, td_vnet_lpush = 0x0, td_intr_frame = 0x0} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_umtx.c =================================================================== --- /usr/src/sys/kern/kern_umtx.c (revision 227550) +++ /usr/src/sys/kern/kern_umtx.c (working copy) @@ -2441,7 +2441,7 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw uint32_t flags, wrflags; int32_t state, oldstate; int32_t blocked_readers; - int error; + int error, loops; uq = td->td_umtxq; flags = fuword32(&rwlock->rw_flags); @@ -2453,6 +2453,7 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw if (!(fflag & URWLOCK_PREFER_READER) && !(flags & URWLOCK_PREFER_READER)) wrflags |= URWLOCK_WRITE_WAITERS; + loops = 0; for (;;) { state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state)); /* try to lock it */ @@ -2496,6 +2497,10 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); + if (loops++) { + umtx_key_release(&uq->uq_key); + return (EINVAL); + } continue; } @@ -2577,7 +2582,7 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw int32_t state, oldstate; int32_t blocked_writers; int32_t blocked_readers; - int error; + int error, loops; uq = td->td_umtxq; flags = fuword32(&rwlock->rw_flags); @@ -2586,6 +2591,7 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw return (error); blocked_readers = 0; + loops = 0; for (;;) { state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state)); while (!(state & URWLOCK_WRITE_OWNER) && URWLOCK_READER_COUNT(state) == 0) { @@ -2606,7 +2612,6 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); } - break; } @@ -2633,6 +2638,10 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); + if (loops++) { + umtx_key_release(&uq->uq_key); + return (EINVAL); + } continue; } sleep: -- Test scenario: syscall2.sh