GENERIC HEAD from 2020-02-12 14:06:02 UTC, r357822M, vmcore.1182 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #2 r357822M: Fri Feb 14 09:30:37 CET 2020 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 9.0.0 (tags/RELEASE_900/final 372316) (based on LLVM 9.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.01-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 3522170880 (3359 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... Root mount waiting for: CAMWARNING: WITNESS option enabled, expect reduced performance. usbus0WARNING: DIAGNOSTIC option enabled, expect reduced performance. usbus1ugen4.1: at usbus4 ugen5.1: at usbus5 ugen2.1: at usbus2 ugen1.1: at usbus1 uhub0 on usbus4 uhub1 on usbus1 uhub2 on usbus5 uhub1: on usbus1 uhub2: on usbus5 uhub3 on usbus2 uhub0: on usbus4 uhub3: on usbus2 usbus2ugen3.1: at usbus3 ugen0.1: at usbus0 usbus3uhub4 usbus4 usbus5 on usbus3 Expensive timeout(9) function: 0xa3de50(0x17eb9000) 0.009383036 s uhub5 (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 on usbus0 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout uhub4: on usbus3 (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted uhub5: on usbus0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA8-ACS SATA 2.x device ada0: Serial Number 9VMJ99G5 ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors) uhub3: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub2: 10 ports with 10 removable, self powered mountroot: waiting for device /dev/ada0s1a... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009952 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 605461 free (2221 frags, 75405 blocks, 0.2% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 721567 free (10951 frags, 88827 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 33220949 free (26309 frags, 4149330 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 900182 free (294958 frags, 75653 blocks, 2.9% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 5212897 free (163809 frags, 631136 blocks, 0.6% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Feb 14 09:34:50 x4 smartd[622]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Feb 14 09:34:50 x4 smartd[622]: see the following Seagate web pages: Feb 14 09:34:50 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Feb 14 09:34:50 x4 smartd[622]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Fri Feb 14 09:34 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Feb 14 09:42:43 x4 su[911]: pho to root on /dev/pts/0 20200214 09:42:58 all (1/1): syscall4.sh lock order reversal: (sleepable after non-sleepable) 1st 0x9d15604 seqflq (seqflq) @ dev/sound/midi/sequencer.c:757 2nd 0x1bfb21c midistat lock (midistat lock) @ dev/sound/midi/midi.c:1491 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfdd53d at _sx_xlock+0x4d #3 0xdc556a at midimapper_open+0x1a #4 0xdc7f1d at mseq_open+0xdd #5 0xea6900 at devfs_open+0x100 #6 0x142c6ed at VOP_OPEN_APV+0x4d #7 0x10ab7a4 at vn_open_vnode+0x184 #8 0x10ab19e at vn_open_cred+0x39e #9 0x10aadf0 at vn_open+0x20 #10 0x10a34fc at kern_openat+0x23c #11 0x10a376f at sys_openat+0x2f #12 0x1419016 at syscall+0x306 #13 0xffc033c9 at exit_tag+0xd8bfbcd9 lock order reversal: 1st 0x22eafa44 ufs (ufs) @ kern/vfs_mount.c:1628 2nd 0x253b4314 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1404 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaa12 at lockmgr_lock_fast_path+0x132 #3 0x1085e44 at vop_stdlock+0x24 #4 0x142eaf8 at VOP_LOCK1_APV+0x38 #5 0x10ab9cf at _vn_lock+0x4f #6 0x12a91cb at ffs_flushfiles+0x9b #7 0x128de0b at softdep_flushfiles+0x9b #8 0x12ab862 at ffs_unmount+0x72 #9 0x108fce8 at dounmount+0x3f8 #10 0x108f88d at kern_unmount+0x25d #11 0x108f623 at sys_unmount+0x13 #12 0x1419016 at syscall+0x306 #13 0xffc033c9 at exit_tag+0xd8bfbcd9 20200214 09:45:12 all (1/1): syscall4.sh mount leak: 1/98 20200214 09:47:22 all (1/1): syscall4.sh mount leak: 1/99 20200214 09:49:29 all (1/1): syscall4.sh mount leak: 1/100 20200214 09:51:35 all (1/1): syscall4.sh Feb 14 09:51:51 x4 kernel: pid 66139 (swap), jid 0, uid 0, was killed: out of swap space lock order reversal: 1st 0x1e22300 cpslk (cpslk) @ i386/i386/copyout.c:121 2nd 0x18ce621c bufwait (bufwait) @ vm/vm_pager.c:391 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1035cca at witness_checkorder+0x90a #2 0xfaaf3d at lockmgr_xlock_hard+0x4d #3 0xfabb7c at __lockmgr_args+0x6bc #4 0x12f1e39 at pbuf_ctor+0x79 #5 0x12c7bd0 at item_ctor+0x330 #6 0x12c661a at zone_alloc_item+0x14a #7 0x12c8b41 at uma_zalloc_single+0x21 #8 0x12c714a at uma_zalloc_arg+0xda #9 0x12c0167 at swap_pager_getpages+0x2e7 #10 0x12f2064 at vm_pager_get_pages+0x44 #11 0x12cfba2 at vm_fault+0x7b2 #12 0x12cf2d2 at vm_fault_trap+0x52 #13 0x1418a11 at trap_pfault+0x161 #14 0x1417fb8 at trap+0x3a8 #15 0xffc0316f at exit_tag+0xd8bfba7f #16 0x28 at ??+0 Feb 14 09:52:01 x4 kernel: pid 66135 (swap), jid 0, uid 0, was killed: out of swap space mount leak: 1/101 20200214 09:53:55 all (1/1): syscall4.sh Feb 14 09:54:23 x4 kernel: pid 70283 (swap), jid 0, uid 0, was killed: out of swap space Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex semid (semid) r = 0 (0x187d9c00) locked @ kern/sysv_sem.c:757 exclusive sleep mutex sem (sem) r = 0 (0x1dfe794) locked @ kern/sysv_sem.c:756 stack backtrace: #0 0x1035ed1 at witness_debugger+0x61 #1 0x1036c85 at witness_warn+0x355 #2 0x141890a at trap_pfault+0x5a #3 0x1417fb8 at trap+0x3a8 #4 0xffc0316f at exit_tag+0xd8bfba7f #5 0x28 at ??+0 Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x187d6000 fault code = supervisor read data, page not present instruction pointer = 0x20:0x1046736 stack pointer = 0x28:0x278a4a98 frame pointer = 0x28:0x278a4ad8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 70635 (ipcrm) trap number = 12 panic: page fault cpuid = 0 time = 1581670472 KDB: stack backtrace: db_trace_self_wrapper(e643cc,1aa7e60,0,278a4894,bd5380,...) at db_trace_self_wrapper+0x2a/frame 0x278a4868 kdb_backtrace(e,38647830,ad71745e,278a4a58,278a4a58,...) at kdb_backtrace+0x2e/frame 0x278a48c8 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x11f/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- KDB: enter: panic [ thread pid 70635 tid 100614 ] Stopped at kdb_enter+0x35: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 70635 tid 100614 td 0x27300e00 kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- db:1:pho1> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x7eaec0 curthread = 0x27300e00: pid 70635 tid 100614 critnest 1 "ipcrm" curpcb = 0x278a4d40 fpcurthread = 0x27300e00: pid 70635 "ipcrm" idlethread = 0x9df8e00: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x278a4d30 common_tssp = 0xffc01260 tlb gen = 3375207 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x8145ec0 curthread = 0x22fb5700: pid 70288 tid 100218 critnest 0 "swap" curpcb = 0x27868d40 fpcurthread = 0x22fb5700: pid 70288 "swap" idlethread = 0x9df8a80: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x27868d30 common_tssp = 0xffc012c8 tlb gen = 3375207 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x814bec0 curthread = 0x9985380: pid 7 tid 100043 critnest 0 "doneq0" curpcb = 0xb684d40 fpcurthread = none idlethread = 0x9df8700: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0xb684d30 common_tssp = 0xffc01330 tlb gen = 3375207 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x8151ec0 curthread = 0x27289880: pid 70286 tid 101028 critnest 0 "swap" curpcb = 0x26f9cd40 fpcurthread = 0x27289880: pid 70286 "swap" idlethread = 0x9df8380: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x26f9cd30 common_tssp = 0xffc01398 tlb gen = 3375205 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 70635 (ipcrm) thread 0x27300e00 (100614) exclusive sleep mutex semid (semid) r = 0 (0x187d9c00) locked @ kern/sysv_sem.c:757 exclusive sleep mutex sem (sem) r = 0 (0x1dfe794) locked @ kern/sysv_sem.c:756 Process 394 (syslogd) thread 0x18871c00 (100128) exclusive lockmgr ufs (ufs) r = 0 (0x205ec5f4) locked @ kern/vfs_syscalls.c:3431 Process 21 (pagedaemon) thread 0x181caa80 (100101) exclusive sx umareclaim (umareclaim) r = 0 (0x1e1395c) locked @ vm/uma_core.c:4666 Process 7 (cam) thread 0x9985380 (100043) exclusive sleep mutex vm laundry pagequeue (vm pagequeue) r = 0 (0x1e4c9c0) locked @ vm/vm_page.c:3290 exclusive rw vm object (vm object) r = 0 (0x2674d3d8) locked @ vm/swap_pager.c:1587 db:1:pho1> show lockedvnods Locked vnodes vnode 0x205ec5c0: type VREG usecount 1, writecount 1, refcount 13 flags (VMP_LAZYLIST) v_object 0x2048a290 ref 0 pages 44 cleanbuf 11 dirtybuf 0 lock type ufs: EXCL by thread 0x18871c00 (pid 394, syslogd, tid 100128) #0 0xfab2a5 at lockmgr_xlock_hard+0x3b5 #1 0xfabb7c at __lockmgr_args+0x6bc #2 0x12ada8f at ffs_lock+0x6f #3 0x142eaf8 at VOP_LOCK1_APV+0x38 #4 0x10ab9cf at _vn_lock+0x4f #5 0x10a777c at kern_fsync+0xfc #6 0x10a78d2 at sys_fsync+0x12 #7 0x1419016 at syscall+0x306 #8 0xffc033c9 at exit_tag+0xd8bfbcd9 nlink=1, effnlink=1, size=207552, extsize 0 generation=d7118cae, uid=0, gid=0, flags=0x0 ino 6052897, on dev ada0s1e db:1:pho1> show allchains chain 1: thread 100305 (pid 70638, sed) sleeping on 0x1fc21350 "piperd" chain 2: thread 100885 (pid 70637, sh) sleeping on 0x1afa800 "pfault" chain 3: thread 101081 (pid 70636, sh) sleeping on 0x237a53f8 "wait" chain 4: thread 100418 (pid 70612, xargs) sleeping on 0x18dffaa0 "wait" chain 5: thread 100260 (pid 70560, sh) sleeping on 0x1fc211a8 "piperd" chain 6: thread 100240 (pid 70296, swap) sleeping on 0x1afa800 "pfault" chain 7: thread 101375 (pid 70294, swap) sleeping on 0x2614f4fc "swread" chain 8: thread 100309 (pid 70293, swap) sleeping on 0x1afa800 "pfault" chain 9: thread 100441 (pid 70289, swap) sleeping on 0x1afa800 "pfault" chain 10: thread 100241 (pid 70287, swap) sleeping on 0x1afa800 "pfault" chain 11: thread 100258 (pid 70285, swap) sleeping on 0x1afa800 "pfault" chain 12: thread 100189 (pid 70284, swap) sleeping on 0x1afa800 "pfault" chain 13: thread 100643 (pid 70282, swap) sleeping on 0x1afa800 "pfault" chain 14: thread 100239 (pid 70281, swap) blocked on lockmgr 37'7'x#x#QSHARED (count 1384152) chain 15: thread 100959 (pid 70273, swap) sleeping on 0x1d921fe "nanslp" chain 16: thread 101350 (pid 70267, md10) sleeping on 0x23045000 "mdwait" chain 17: thread 100577 (pid 70226, sh) sleeping on 0x26bc30a4 "wait" chain 18: thread 100323 (pid 70225, tee) sleeping on 0x2048f4f8 "piperd" chain 19: thread 100391 (pid 70224, sh) blocked on lockmgr )&T#7'SHARED (count 1384152) chain 20: thread 100109 (pid 2313, aiod4) sleeping on 0x1885a9fc "aiordy" chain 21: thread 100209 (pid 2312, aiod3) sleeping on 0x26b129fc "aiordy" chain 22: thread 100179 (pid 2311, aiod2) blocked on lockmgr &QSHARED (count 1384152) chain 23: thread 100219 (pid 2310, aiod1) sleeping on 0x1886174c "aiordy" chain 24: thread 100111 (pid 1069, sh) sleeping on 0x20314000 "piperd" chain 25: thread 100175 (pid 1068, awk) sleeping on 0x1fc59d40 "piperd" chain 26: thread 100210 (pid 1067, sh) sleeping on 0x26b126a8 "wait" chain 27: thread 100182 (pid 1046, sh) blocked on lockmgr )z# s s QSHARED (count 1384152) chain 28: thread 100197 (pid 949, sh) sleeping on 0x237a30a4 "wait" chain 29: thread 100199 (pid 913, bash) sleeping on 0x237a29fc "wait" chain 30: thread 100203 (pid 911, csh) sleeping on 0x18da03b8 "pause" chain 31: thread 100202 (pid 910, su) sleeping on 0x237a2000 "wait" chain 32: thread 100204 (pid 901, bash) sleeping on 0x18da0000 "wait" chain 33: thread 100200 (pid 900, sshd) sleeping on 0x17edfee4 "select" chain 34: thread 100198 (pid 898, sshd) sleeping on 0x9df4a24 "select" chain 35: thread 100196 (pid 770, getty) sleeping on 0x18202c74 "ttyin" chain 36: thread 100195 (pid 769, getty) sleeping on 0x1887d474 "ttyin" chain 37: thread 100194 (pid 768, getty) sleeping on 0x1887d874 "ttyin" chain 38: thread 100193 (pid 767, getty) sleeping on 0x1887dc74 "ttyin" chain 39: thread 100192 (pid 766, getty) sleeping on 0x1887e074 "ttyin" chain 40: thread 100191 (pid 765, getty) sleeping on 0x1887e474 "ttyin" chain 41: thread 100190 (pid 764, getty) sleeping on 0x1887e874 "ttyin" chain 42: thread 100122 (pid 763, getty) sleeping on 0x1887ec74 "ttyin" chain 43: thread 100112 (pid 762, getty) sleeping on 0x1887f074 "ttyin" chain 44: thread 100183 (pid 745, inetd) sleeping on 0x1837b4a4 "select" chain 45: thread 100121 (pid 708, nfscbd: master) sleeping on 0x20545750 "rpcsvc" chain 46: thread 100184 (pid 708, nfscbd: service) sleeping on 0x18d0e950 "rpcsvc" chain 47: thread 100185 (pid 708, nfscbd: service) sleeping on 0x18d0e8d0 "rpcsvc" chain 48: thread 100186 (pid 708, nfscbd: service) sleeping on 0x18d0e850 "rpcsvc" chain 49: thread 100181 (pid 707, nfscbd) sleeping on 0x204d2b68 "accept" chain 50: thread 100171 (pid 674, cron) sleeping on 0x1d921fe "nanslp" chain 51: thread 100176 (pid 670, sendmail) sleeping on 0x18da1108 "pause" chain 52: thread 100177 (pid 667, sendmail) sleeping on 0x9df42a4 "select" chain 53: thread 100173 (pid 664, sshd) sleeping on 0x1837b564 "select" chain 54: thread 100129 (pid 624, smartd) sleeping on 0x1d921fe "nanslp" chain 55: thread 100126 (pid 600, powerd) sleeping on 0x9df43e4 "select" chain 56: thread 100170 (pid 597, ntpd) sleeping on 0x18cef5e4 "select" chain 57: thread 100136 (pid 559, NLM: master) sleeping on 0x205458d0 "rpcsvc" chain 58: thread 100169 (pid 556, rpc.statd) sleeping on 0x1837b6a4 "select" chain 59: thread 100137 (pid 555, nfsd: master) sleeping on 0x20545950 "rpcsvc" chain 60: thread 100138 (pid 555, nfsd: service) sleeping on 0x1888c750 "rpcsvc" chain 61: thread 100139 (pid 555, nfsd: service) sleeping on 0x1888c6d0 "rpcsvc" chain 62: thread 100140 (pid 555, nfsd: service) sleeping on 0x18d0fe50 "rpcsvc" chain 63: thread 100141 (pid 555, nfsd: service) sleeping on 0x1888c650 "rpcsvc" chain 64: thread 100142 (pid 555, nfsd: service) sleeping on 0x1888c5d0 "rpcsvc" chain 65: thread 100143 (pid 555, nfsd: service) sleeping on 0x1888c550 "rpcsvc" chain 66: thread 100144 (pid 555, nfsd: service) sleeping on 0x1888c4d0 "rpcsvc" chain 67: thread 100145 (pid 555, nfsd: service) sleeping on 0x1888c7d0 "rpcsvc" chain 68: thread 100146 (pid 555, nfsd: service) sleeping on 0x1888c450 "rpcsvc" chain 69: thread 100147 (pid 555, nfsd: service) sleeping on 0x1888c3d0 "rpcsvc" chain 70: thread 100148 (pid 555, nfsd: service) sleeping on 0x9933f50 "rpcsvc" chain 71: thread 100149 (pid 555, nfsd: service) sleeping on 0x1888c350 "rpcsvc" chain 72: thread 100150 (pid 555, nfsd: service) sleeping on 0x1888c2d0 "rpcsvc" chain 73: thread 100151 (pid 555, nfsd: service) sleeping on 0x9933ed0 "rpcsvc" chain 74: thread 100152 (pid 555, nfsd: service) sleeping on 0x1888c250 "rpcsvc" chain 75: thread 100153 (pid 555, nfsd: service) sleeping on 0x9933e50 "rpcsvc" chain 76: thread 100154 (pid 555, nfsd: service) sleeping on 0x1888c1d0 "rpcsvc" chain 77: thread 100155 (pid 555, nfsd: service) sleeping on 0x9933dd0 "rpcsvc" chain 78: thread 100156 (pid 555, nfsd: service) sleeping on 0x1888c150 "rpcsvc" chain 79: thread 100157 (pid 555, nfsd: service) sleeping on 0x9933d50 "rpcsvc" chain 80: thread 100158 (pid 555, nfsd: service) sleeping on 0x1888c0d0 "rpcsvc" chain 81: thread 100159 (pid 555, nfsd: service) sleeping on 0x1888c050 "rpcsvc" chain 82: thread 100160 (pid 555, nfsd: service) sleeping on 0x9933cd0 "rpcsvc" chain 83: thread 100161 (pid 555, nfsd: service) sleeping on 0x1888bf50 "rpcsvc" chain 84: thread 100162 (pid 555, nfsd: service) sleeping on 0x9933c50 "rpcsvc" chain 85: thread 100163 (pid 555, nfsd: service) sleeping on 0x1888bed0 "rpcsvc" chain 86: thread 100164 (pid 555, nfsd: service) sleeping on 0x1888be50 "rpcsvc" chain 87: thread 100165 (pid 555, nfsd: service) sleeping on 0x9933bd0 "rpcsvc" chain 88: thread 100166 (pid 555, nfsd: service) sleeping on 0x1888bdd0 "rpcsvc" chain 89: thread 100167 (pid 555, nfsd: service) sleeping on 0x9933b50 "rpcsvc" chain 90: thread 100168 (pid 555, nfsd: service) sleeping on 0x9933ad0 "rpcsvc" chain 91: thread 100135 (pid 552, nfsd) sleeping on 0x18cef624 "select" chain 92: thread 100130 (pid 546, mountd) sleeping on 0x9df4464 "select" chain 93: thread 100110 (pid 539, nfsuserd) sleeping on 0x9df4424 "select" chain 94: thread 100134 (pid 538, nfsuserd) sleeping on 0x9df5724 "select" chain 95: thread 100120 (pid 537, nfsuserd) sleeping on 0x18cefda4 "select" chain 96: thread 100132 (pid 536, nfsuserd) sleeping on 0x9df59a4 "select" chain 97: thread 100124 (pid 535, nfsuserd) sleeping on 0x181893b8 "pause" chain 98: thread 100107 (pid 492, watchdogd) sleeping on 0x1d921fc "nanslp" chain 99: thread 100114 (pid 403, rpcbind) sleeping on 0x18cf01a4 "select" chain 100: thread 100128 (pid 394, syslogd) sleeping on 0x205ec6d0 "bo_wwait" chain 101: thread 100113 (pid 322, devd) sleeping on 0x17ee0ea4 "select" chain 102: thread 100104 (pid 25, vnlru) sleeping on 0x1885b0a4 "vlruwt" chain 103: thread 100103 (pid 24, syncer) sleeping on 0x1dff128 "syncer" chain 104: thread 100099 (pid 23, bufdaemon) sleeping on 0x1dfeaec "qsleep" chain 105: thread 100102 (pid 23, bufspacedaemon-0) sleeping on 0x1acc040 "-" chain 106: thread 100115 (pid 23, /home worker) sleeping on 0x1fdae084 "sdflush" chain 107: thread 100116 (pid 23, /tmp worker) sleeping on 0x1fdae484 "sdflush" chain 108: thread 100117 (pid 23, /usr worker) sleeping on 0x9966684 "sdflush" chain 109: thread 100118 (pid 23, /var worker) sleeping on 0x1fdae284 "sdflush" chain 110: thread 100671 (pid 23, /mnt worker) sleeping on 0x18d7f284 "sdflush" chain 111: thread 100098 (pid 22, vmdaemon) sleeping on 0x1e13e30 "psleep" chain 112: thread 100097 (pid 21, dom0) sleeping on 0x187ab55 "pwait" chain 113: thread 100100 (pid 21, laundry: dom0) sleeping on 0x1e13880 "swbufa" chain 114: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1bf607c "-" chain 115: thread 100090 (pid 19, sctp_iterator) sleeping on 0x1e5dd78 "waiting_for_work" chain 116: thread 100089 (pid 18, soaiod4) sleeping on 0x1dfe728 "-" chain 117: thread 100088 (pid 17, soaiod3) sleeping on 0x1dfe728 "-" chain 118: thread 100087 (pid 16, soaiod2) sleeping on 0x1dfe728 "-" chain 119: thread 100086 (pid 9, soaiod1) sleeping on 0x1dfe728 "-" chain 120: thread 100083 (pid 8, fdc0) sleeping on 0x1813023c "-" chain 121: thread 100048 (pid 15, usbus0) sleeping on 0x17ec2aac "-" chain 122: thread 100049 (pid 15, usbus0) sleeping on 0x17ec2adc "-" chain 123: thread 100050 (pid 15, usbus0) sleeping on 0x17ec2b0c "-" chain 124: thread 100051 (pid 15, usbus0) sleeping on 0x17ec2b3c "-" chain 125: thread 100052 (pid 15, usbus0) sleeping on 0x17ec2b6c "-" chain 126: thread 100054 (pid 15, usbus1) sleeping on 0x18197aac "-" chain 127: thread 100055 (pid 15, usbus1) sleeping on 0x18197adc "-" chain 128: thread 100056 (pid 15, usbus1) sleeping on 0x18197b0c "-" chain 129: thread 100057 (pid 15, usbus1) sleeping on 0x18197b3c "-" chain 130: thread 100058 (pid 15, usbus1) sleeping on 0x18197b6c "-" chain 131: thread 100060 (pid 15, usbus2) sleeping on 0x1819faac "-" chain 132: thread 100061 (pid 15, usbus2) sleeping on 0x1819fadc "-" chain 133: thread 100062 (pid 15, usbus2) sleeping on 0x1819fb0c "-" chain 134: thread 100063 (pid 15, usbus2) sleeping on 0x1819fb3c "-" chain 135: thread 100064 (pid 15, usbus2) sleeping on 0x1819fb6c "-" chain 136: thread 100065 (pid 15, usbus3) sleeping on 0x181a7aac "-" chain 137: thread 100066 (pid 15, usbus3) sleeping on 0x181a7adc "-" chain 138: thread 100067 (pid 15, usbus3) sleeping on 0x181a7b0c "-" chain 139: thread 100068 (pid 15, usbus3) sleeping on 0x181a7b3c "-" chain 140: thread 100069 (pid 15, usbus3) sleeping on 0x181a7b6c "-" chain 141: thread 100070 (pid 15, usbus4) sleeping on 0x181abaac "-" chain 142: thread 100071 (pid 15, usbus4) sleeping on 0x181abadc "-" chain 143: thread 100072 (pid 15, usbus4) sleeping on 0x181abb0c "-" chain 144: thread 100073 (pid 15, usbus4) sleeping on 0x181abb3c "-" chain 145: thread 100074 (pid 15, usbus4) sleeping on 0x181abb6c "-" chain 146: thread 100076 (pid 15, usbus5) sleeping on 0x17ebdc84 "-" chain 147: thread 100077 (pid 15, usbus5) sleeping on 0x17ebdcb4 "-" chain 148: thread 100078 (pid 15, usbus5) sleeping on 0x17ebdce4 "-" chain 149: thread 100079 (pid 15, usbus5) sleeping on 0x17ebdd14 "-" chain 150: thread 100080 (pid 15, usbus5) sleeping on 0x17ebdd44 "-" chain 151: thread 100096 (pid 7, scanner) sleeping on 0x1b2f468 "-" chain 152: thread 100042 (pid 6, crypto returns 3) sleeping on 0x9906dc4 "crypto_ret_wait" chain 153: thread 100041 (pid 5, crypto returns 2) sleeping on 0x9906d8c "crypto_ret_wait" chain 154: thread 100040 (pid 4, crypto returns 1) sleeping on 0x9906d54 "crypto_ret_wait" chain 155: thread 100039 (pid 3, crypto returns 0) sleeping on 0x9906d1c "crypto_ret_wait" chain 156: thread 100038 (pid 2, crypto) sleeping on 0x1e12b28 "crypto_wait" chain 157: thread 100033 (pid 14, sequencer 00) sleeping on 0x9d1564c "seqstate" chain 158: thread 100030 (pid 13, g_event) sleeping on 0x1e4c2f0 "-" chain 159: thread 100031 (pid 13, g_up) sleeping on 0x1e4c2f8 "-" chain 160: thread 100032 (pid 13, g_down) sleeping on 0x1e4c2f4 "-" chain 161: thread 100002 (pid 1, init) sleeping on 0x9df10a4 "wait" chain 162: thread 100001 (pid 10, audit) sleeping on 0x1e5e50c "audit_worker_cv" chain 163: thread 100000 (pid 0, swapper) sleeping on 0x1e21ce0 "swapin" chain 164: thread 100008 (pid 0, thread taskq) sleeping on 0x9d3b600 "-" chain 165: thread 100010 (pid 0, config_0) sleeping on 0x9d36e00 "-" chain 166: thread 100011 (pid 0, kqueue_ctx taskq) sleeping on 0x9d3b100 "-" chain 167: thread 100012 (pid 0, aiod_kick taskq) sleeping on 0x9d3ae00 "-" chain 168: thread 100014 (pid 0, softirq_0) sleeping on 0x9d36d00 "-" chain 169: thread 100015 (pid 0, softirq_1) sleeping on 0x9d36c00 "-" chain 170: thread 100016 (pid 0, softirq_2) sleeping on 0x9d36b00 "-" chain 171: thread 100017 (pid 0, softirq_3) sleeping on 0x9d36a00 "-" chain 172: thread 100018 (pid 0, if_io_tqg_0) sleeping on 0x9d36900 "-" chain 173: thread 100019 (pid 0, if_io_tqg_1) sleeping on 0x9d36800 "-" chain 174: thread 100020 (pid 0, if_io_tqg_2) sleeping on 0x9d36700 "-" chain 175: thread 100021 (pid 0, if_io_tqg_3) sleeping on 0x9d36600 "-" chain 176: thread 100022 (pid 0, if_config_tqg_0) sleeping on 0x9d36500 "-" chain 177: thread 100029 (pid 0, firmware taskq) sleeping on 0x9d39a00 "-" chain 178: thread 100034 (pid 0, crypto_0) sleeping on 0x9d39400 "-" chain 179: thread 100035 (pid 0, crypto_1) sleeping on 0x9d39400 "-" chain 180: thread 100036 (pid 0, crypto_2) sleeping on 0x9d39400 "-" chain 181: thread 100037 (pid 0, crypto_3) sleeping on 0x9d39400 "-" chain 182: thread 100085 (pid 0, mca taskq) sleeping on 0x17ee5400 "-" chain 183: thread 100091 (pid 0, acpi_task_0) sleeping on 0x18377c00 "-" chain 184: thread 100092 (pid 0, acpi_task_1) sleeping on 0x18377c00 "-" chain 185: thread 100093 (pid 0, acpi_task_2) sleeping on 0x18377c00 "-" chain 186: thread 100095 (pid 0, CAM taskq) sleeping on 0x9d38900 "-" db:1:pho1> show mount 0x18bfb580 /dev/ada0s1a on / (ufs) 0x18bfc040 devfs on /dev (devfs) 0x18bfaac0 /dev/ada0s1f on /home (ufs) 0x1fdcf000 /dev/label/tmp on /tmp (ufs) 0x18bfe080 /dev/ada0s1d on /usr (ufs) 0x18bfd5c0 /dev/ada0s1e on /var (ufs) 0x18bfcb00 procfs on /proc (procfs) 0x1fdd25c0 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 466 Buf domain 0 freebufs 466 lofreebufs 311 hifreebufs 466 bufspace 82376704 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 7 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5245 (81297408) dirtyq count 8 (122880) wakeup 0 lim 28 CPU 15, 20, 10, 15, Locked buffers: 1 space 16384 Total buffers: 5706 space 82376704 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 70638 70636 1046 0 S piperd 0x1fc21350 sed 70637 70636 1046 0 D pfault 0x1afa800 sh 70636 70560 1046 0 S wait 0x237a53f8 sh 70635 70612 949 0 R+ CPU 0 ipcrm 70612 70226 949 0 S+ wait 0x18dffaa0 xargs 70560 1067 1046 0 S piperd 0x1fc211a8 sh 70298 70281 70273 0 R swap 70296 70281 70273 0 D pfault 0x1afa800 swap 70295 70281 70273 0 R swap 70294 70281 70273 0 D swread 0x2614f4fc swap 70293 70281 70273 0 D pfault 0x1afa800 swap 70292 70281 70273 0 R swap 70291 70281 70273 0 R swap 70290 70281 70273 0 R swap 70289 70281 70273 0 D pfault 0x1afa800 swap 70288 70281 70273 0 R CPU 1 swap 70287 70281 70273 0 D pfault 0x1afa800 swap 70286 70281 70273 0 R CPU 3 swap 70285 70281 70273 0 D pfault 0x1afa800 swap 70284 70281 70273 0 D pfault 0x1afa800 swap 70282 70281 70273 0 D pfault 0x1afa800 swap 70281 70273 70273 0 SW wait 0x273706a8 swap 70273 1 70273 0 Ss nanslp 0x1d921fe swap 70267 0 0 0 DL mdwait 0x23045000 [md10] 70226 70224 949 0 SW+ wait 0x26bc30a4 sh 70225 949 949 0 S+ piperd 0x2048f4f8 tee 70224 949 949 0 SW+ wait 0x27372354 sh 2313 0 0 0 DL aiordy 0x1885a9fc [aiod4] 2312 0 0 0 DL aiordy 0x26b129fc [aiod3] 2311 0 0 0 DL aiordy 0x18da26a8 [aiod2] 2310 0 0 0 DL aiordy 0x1886174c [aiod1] 1069 1046 1046 0 S piperd 0x20314000 sh 1068 1046 1046 0 S piperd 0x1fc59d40 awk 1067 1046 1046 0 SW wait 0x26b126a8 sh 1046 1 1046 0 SWs wait 0x18da09fc sh 949 913 949 0 SW+ wait 0x237a30a4 sh 913 911 913 0 SW+ wait 0x237a29fc bash 911 910 911 0 SW+ pause 0x18da03b8 csh 910 901 910 1001 SW+ wait 0x237a2000 su 901 900 901 1001 SWs+ wait 0x18da0000 bash 900 898 898 1001 S select 0x17edfee4 sshd 898 664 898 0 Ss select 0x9df4a24 sshd 770 1 770 0 Ss+ ttyin 0x18202c74 getty 769 1 769 0 Ss+ ttyin 0x1887d474 getty 768 1 768 0 Ss+ ttyin 0x1887d874 getty 767 1 767 0 Ss+ ttyin 0x1887dc74 getty 766 1 766 0 Ss+ ttyin 0x1887e074 getty 765 1 765 0 Ss+ ttyin 0x1887e474 getty 764 1 764 0 Ss+ ttyin 0x1887e874 getty 763 1 763 0 Ss+ ttyin 0x1887ec74 getty 762 1 762 0 Ss+ ttyin 0x1887f074 getty 745 1 745 0 Ss select 0x1837b4a4 inetd 708 707 707 0 S (threaded) nfscbd 100121 S rpcsvc 0x20545750 nfscbd: master 100184 S rpcsvc 0x18d0e950 nfscbd: service 100185 S rpcsvc 0x18d0e8d0 nfscbd: service 100186 S rpcsvc 0x18d0e850 nfscbd: service 707 1 707 0 SWs accept 0x204d2b68 nfscbd 674 1 674 0 SWs nanslp 0x1d921fe cron 670 1 670 25 SWs pause 0x18da1108 sendmail 667 1 667 0 Ss select 0x9df42a4 sendmail 664 1 664 0 Ss select 0x1837b564 sshd 624 1 623 0 SW nanslp 0x1d921fe smartd 600 1 600 0 Ss select 0x9df43e4 powerd 597 1 597 0 Ss (threaded) ntpd 100170 S select 0x18cef5e4 ntpd 559 1 559 0 Ss rpcsvc 0x205458d0 NLM: master 556 1 556 0 Ss select 0x1837b6a4 rpc.statd 555 552 552 0 S (threaded) nfsd 100137 S rpcsvc 0x20545950 nfsd: master 100138 S rpcsvc 0x1888c750 nfsd: service 100139 S rpcsvc 0x1888c6d0 nfsd: service 100140 S rpcsvc 0x18d0fe50 nfsd: service 100141 S rpcsvc 0x1888c650 nfsd: service 100142 S rpcsvc 0x1888c5d0 nfsd: service 100143 S rpcsvc 0x1888c550 nfsd: service 100144 S rpcsvc 0x1888c4d0 nfsd: service 100145 S rpcsvc 0x1888c7d0 nfsd: service 100146 S rpcsvc 0x1888c450 nfsd: service 100147 S rpcsvc 0x1888c3d0 nfsd: service 100148 S rpcsvc 0x9933f50 nfsd: service 100149 S rpcsvc 0x1888c350 nfsd: service 100150 S rpcsvc 0x1888c2d0 nfsd: service 100151 S rpcsvc 0x9933ed0 nfsd: service 100152 S rpcsvc 0x1888c250 nfsd: service 100153 S rpcsvc 0x9933e50 nfsd: service 100154 S rpcsvc 0x1888c1d0 nfsd: service 100155 S rpcsvc 0x9933dd0 nfsd: service 100156 S rpcsvc 0x1888c150 nfsd: service 100157 S rpcsvc 0x9933d50 nfsd: service 100158 S rpcsvc 0x1888c0d0 nfsd: service 100159 S rpcsvc 0x1888c050 nfsd: service 100160 S rpcsvc 0x9933cd0 nfsd: service 100161 S rpcsvc 0x1888bf50 nfsd: service 100162 S rpcsvc 0x9933c50 nfsd: service 100163 S rpcsvc 0x1888bed0 nfsd: service 100164 S rpcsvc 0x1888be50 nfsd: service 100165 S rpcsvc 0x9933bd0 nfsd: service 100166 S rpcsvc 0x1888bdd0 nfsd: service 100167 S rpcsvc 0x9933b50 nfsd: service 100168 S rpcsvc 0x9933ad0 nfsd: service 552 1 552 0 Ss select 0x18cef624 nfsd 546 1 546 0 Ss select 0x9df4464 mountd 539 535 535 0 S select 0x9df4424 nfsuserd 538 535 535 0 S select 0x9df5724 nfsuserd 537 535 535 0 S select 0x18cefda4 nfsuserd 536 535 535 0 S select 0x9df59a4 nfsuserd 535 1 535 0 SWs pause 0x181893b8 nfsuserd 492 1 492 0 Ss nanslp 0x1d921fc watchdogd 403 1 403 0 Ss select 0x18cf01a4 rpcbind 394 1 394 0 Ds bo_wwai 0x205ec6d0 syslogd 322 1 322 0 Ss select 0x17ee0ea4 devd 25 0 0 0 DL vlruwt 0x1885b0a4 [vnlru] 24 0 0 0 DL syncer 0x1dff128 [syncer] 23 0 0 0 DL (threaded) [bufdaemon] 100099 D qsleep 0x1dfeaec [bufdaemon] 100102 D - 0x1acc040 [bufspacedaemon-0] 100115 D sdflush 0x1fdae084 [/home worker] 100116 D sdflush 0x1fdae484 [/tmp worker] 100117 D sdflush 0x9966684 [/usr worker] 100118 D sdflush 0x1fdae284 [/var worker] 100671 D sdflush 0x18d7f284 [/mnt worker] 22 0 0 0 DL psleep 0x1e13e30 [vmdaemon] 21 0 0 0 RL (threaded) [pagedaemon] 100097 D pwait 0x187ab55 [dom0] 100100 D swbufa 0x1e13880 [laundry: dom0] 100101 RunQ [uma] 20 0 0 0 DL - 0x1bf607c [rand_harvestq] 19 0 0 0 DL waiting 0x1e5dd78 [sctp_iterator] 18 0 0 0 DL - 0x1dfe728 [soaiod4] 17 0 0 0 DL - 0x1dfe728 [soaiod3] 16 0 0 0 DL - 0x1dfe728 [soaiod2] 9 0 0 0 DL - 0x1dfe728 [soaiod1] 8 0 0 0 DL - 0x1813023c [fdc0] 15 0 0 0 DL (threaded) [usb] 100048 D - 0x17ec2aac [usbus0] 100049 D - 0x17ec2adc [usbus0] 100050 D - 0x17ec2b0c [usbus0] 100051 D - 0x17ec2b3c [usbus0] 100052 D - 0x17ec2b6c [usbus0] 100054 D - 0x18197aac [usbus1] 100055 D - 0x18197adc [usbus1] 100056 D - 0x18197b0c [usbus1] 100057 D - 0x18197b3c [usbus1] 100058 D - 0x18197b6c [usbus1] 100060 D - 0x1819faac [usbus2] 100061 D - 0x1819fadc [usbus2] 100062 D - 0x1819fb0c [usbus2] 100063 D - 0x1819fb3c [usbus2] 100064 D - 0x1819fb6c [usbus2] 100065 D - 0x181a7aac [usbus3] 100066 D - 0x181a7adc [usbus3] 100067 D - 0x181a7b0c [usbus3] 100068 D - 0x181a7b3c [usbus3] 100069 D - 0x181a7b6c [usbus3] 100070 D - 0x181abaac [usbus4] 100071 D - 0x181abadc [usbus4] 100072 D - 0x181abb0c [usbus4] 100073 D - 0x181abb3c [usbus4] 100074 D - 0x181abb6c [usbus4] 100076 D - 0x17ebdc84 [usbus5] 100077 D - 0x17ebdcb4 [usbus5] 100078 D - 0x17ebdce4 [usbus5] 100079 D - 0x17ebdd14 [usbus5] 100080 D - 0x17ebdd44 [usbus5] 7 0 0 0 RL (threaded) [cam] 100043 Run CPU 2 [doneq0] 100096 D - 0x1b2f468 [scanner] 6 0 0 0 DL crypto_ 0x9906dc4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x9906d8c [crypto returns 2] 4 0 0 0 DL crypto_ 0x9906d54 [crypto returns 1] 3 0 0 0 DL crypto_ 0x9906d1c [crypto returns 0] 2 0 0 0 DL crypto_ 0x1e12b28 [crypto] 14 0 0 0 DL seqstat 0x9d1564c [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100030 D - 0x1e4c2f0 [g_event] 100031 D - 0x1e4c2f8 [g_up] 100032 D - 0x1e4c2f4 [g_down] 12 0 0 0 WL (threaded) [intr] 100007 I [swi6: Giant taskq] 100009 I [swi5: fast taskq] 100013 I [swi6: task queue] 100023 I [swi1: netisr 0] 100024 I [swi4: clock (0)] 100025 I [swi4: clock (1)] 100026 I [swi4: clock (2)] 100027 I [swi4: clock (3)] 100028 I [swi3: vm] 100044 I [irq24: hdac0] 100045 I [irq25: re0] 100046 I [irq22: ahci0] 100047 I [irq16: hdac1 ohci0] 100053 I [irq17: ohci1 ohci3] 100059 I [irq18: ohci2 ohci4] 100075 I [irq19: ehci0] 100081 I [irq14: ata0] 100082 I [swi0: uart] 100084 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0x9df10a4 [init] 10 0 0 0 DL audit_w 0x1e5e50c [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x1e21ce0 [swapper] 100008 D - 0x9d3b600 [thread taskq] 100010 D - 0x9d36e00 [config_0] 100011 D - 0x9d3b100 [kqueue_ctx taskq] 100012 D - 0x9d3ae00 [aiod_kick taskq] 100014 D - 0x9d36d00 [softirq_0] 100015 D - 0x9d36c00 [softirq_1] 100016 D - 0x9d36b00 [softirq_2] 100017 D - 0x9d36a00 [softirq_3] 100018 D - 0x9d36900 [if_io_tqg_0] 100019 D - 0x9d36800 [if_io_tqg_1] 100020 D - 0x9d36700 [if_io_tqg_2] 100021 D - 0x9d36600 [if_io_tqg_3] 100022 D - 0x9d36500 [if_config_tqg_0] 100029 D - 0x9d39a00 [firmware taskq] 100034 D - 0x9d39400 [crypto_0] 100035 D - 0x9d39400 [crypto_1] 100036 D - 0x9d39400 [crypto_2] 100037 D - 0x9d39400 [crypto_3] 100085 D - 0x17ee5400 [mca taskq] 100091 D - 0x18377c00 [acpi_task_0] 100092 D - 0x18377c00 [acpi_task_1] 100093 D - 0x18377c00 [acpi_task_2] 100095 D - 0x9d38900 [CAM taskq] 70283 70281 70273 0 Z swap db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000001 05 (000128K) | 000000 04 (000064K) | 000001 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000002 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000000 04 (000064K) | 000000 03 (000032K) | 000000 02 (000016K) | 000000 01 (000008K) | 000000 00 (000004K) | 000000 db:0:pho> acttrace Tracing command ipcrm pid 70635 tid 100614 td 0x27300e00 (CPU 0) kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command swap pid 70288 tid 100218 td 0x22fb5700 (CPU 1) cpustop_handler(200046,ffc0afa8,ffc0af9c,1417ccb,0,...) at cpustop_handler+0x24/frame 0xffc0aed0 ipi_nmi_handler(0,0,0,0,26bc5aa0,...) at ipi_nmi_handler+0x31/frame 0xffc0aee0 trap(ffc0afa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0af9c calltrap() at 0xffc0316f/frame 0xffc0af9c --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70286 tid 101028 td 0x27289880 (CPU 3) cpustop_handler(200046,ffc0dfa8,ffc0df9c,1417ccb,0,...) at cpustop_handler+0x24/frame 0xffc0ded0 ipi_nmi_handler(0,0,0,0,237a4d50,...) at ipi_nmi_handler+0x31/frame 0xffc0dee0 trap(ffc0dfa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0df9c calltrap() at 0xffc0316f/frame 0xffc0df9c --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command cam pid 7 tid 100043 td 0x9985380 (CPU 2) cpustop_handler(46,b6849c0,b6849b4,1417ccb,f,...) at cpustop_handler+0x24/frame 0xb6848e8 ipi_nmi_handler(f,1,54,7fffffff,9deed50,...) at ipi_nmi_handler+0x31/frame 0xb6848f8 trap(b6849c0,8,28,28,1e4c9d0,...) at trap+0xbb/frame 0xb6849b4 calltrap() at 0xffc0316f/frame 0xb6849b4 --- trap 0x13, eip = 0x10364b0, esp = 0xb684a00, ebp = 0xb684a20 --- witness_unlock(1e4c9d0,0,155a962,ce2) at witness_unlock/frame 0xb684a20 vm_page_pqstate_commit(3488310,b684a88,248) at vm_page_pqstate_commit+0x1e0/frame 0xb684a74 vm_page_deactivate_noreuse(3488310,2674d458,2674d3e8,2674d3d8,18cd7380,...) at vm_page_deactivate_noreuse+0x64/frame 0xb684aa0 swp_pager_async_iodone(18cd7380) at swp_pager_async_iodone+0x18d/frame 0xb684ac4 bufdone(18cd7380) at bufdone+0x3e/frame 0xb684b10 swapgeom_done(2342a450) at swapgeom_done+0x43/frame 0xb684b24 g_io_deliver(2342a450,0) at g_io_deliver+0x247/frame 0xb684b60 g_std_done(2342a564) at g_std_done+0x78/frame 0xb684b74 g_io_deliver(2342a564,0) at g_io_deliver+0x247/frame 0xb684bb0 g_std_done(2342a78c) at g_std_done+0x78/frame 0xb684bc4 g_io_deliver(2342a78c,0) at g_io_deliver+0x247/frame 0xb684c00 g_disk_done(2342aac8) at g_disk_done+0xf0/frame 0xb684c34 adadone(9de5700,99c2800) at adadone+0xc6d/frame 0xb684c6c xpt_done_process(0,b684c98,1b2f510,202,9985380,...) at xpt_done_process+0x2f7/frame 0xb684c90 xpt_done_td(1b2f500,b684ce8) at xpt_done_td+0xd5/frame 0xb684cb4 fork_exit(907280,1b2f500,b684ce8,0,0,...) at fork_exit+0x69/frame 0xb684cd4 fork_trampoline() at 0xffc033de/frame 0xb684cd4 --- kthread start db:0:pho> alltrace Tracing command sed pid 70638 tid 100305 td 0x23177180 sched_switch(23177180,104) at sched_switch+0x4f3/frame 0x26e8fa50 mi_switch(104) at mi_switch+0x1c4/frame 0x26e8fa8c sleepq_switch(10c0,5c,0,1da0f94,18dfe9fc,...) at sleepq_switch+0xf7/frame 0x26e8faac sleepq_catch_signals(26e8fb20,fdf10e,1fc21350,5c,1fc214f0,...) at sleepq_catch_signals+0x368/frame 0x26e8fad8 sleepq_wait_sig(1fc21350,5c) at sleepq_wait_sig+0xe/frame 0x26e8fae0 _sleep(1fc21350,1fc214e0,15c,15288cd,0,0,0,0,100) at _sleep+0x27e/frame 0x26e8fb20 pipe_read(18df20e0,26e8fbd8,18d13e00,0,23177180) at pipe_read+0x3bf/frame 0x26e8fb60 dofileread(18df20e0,26e8fbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x26e8fb90 kern_readv(23177180,0,26e8fbd8) at kern_readv+0x3b/frame 0x26e8fbbc sys_read(23177180,2317740c) at sys_read+0x48/frame 0x26e8fc08 syscall(26e8fce8,3b,3b,3b,205f75cc,...) at syscall+0x306/frame 0x26e8fcdc Xint0x80_syscall() at 0xffc033c9/frame 0x26e8fcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 70637 tid 100885 td 0x27302a80 sched_switch(27302a80,104) at sched_switch+0x4f3/frame 0x274d89c8 mi_switch(104) at mi_switch+0x1c4/frame 0x274d8a08 sleepq_switch(27302a80,0,1afa800,274d8a8c,27302a80,...) at sleepq_switch+0xf7/frame 0x274d8a28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x274d8a4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x274d8a8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x274d8ac0 vm_fault(26b321f0,20442000,1,0,0) at vm_fault+0x12f9/frame 0x274d8b98 vm_fault_trap(26b321f0,20442084,1,0,274d8c2c,274d8c30) at vm_fault_trap+0x52/frame 0x274d8bc0 trap_pfault(20442084,274d8c2c,274d8c30) at trap_pfault+0x161/frame 0x274d8c14 trap(274d8ce8,3b,3b,3b,2043e038,...) at trap+0x5c8/frame 0x274d8cdc calltrap() at 0xffc0316f/frame 0x274d8cdc --- trap 0, eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- exit_tag(0,0,0,0,0,...) at 0xffc01430/frame 0xffbfe750 Tracing command sh pid 70636 tid 101081 td 0x2730d380 sched_switch(2730d380,104) at sched_switch+0x4f3/frame 0x2333e9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x2333e9f8 sleepq_switch(2ac0,6c,0,1da2994,237a53f8,...) at sleepq_switch+0xf7/frame 0x2333ea18 sleepq_catch_signals(2333ea8c,fdf10e,237a53f8,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x2333ea44 sleepq_wait_sig(237a53f8,6c) at sleepq_wait_sig+0xe/frame 0x2333ea4c _sleep(237a53f8,237a54a0,36c,1544874,0,0,0,0,100) at _sleep+0x27e/frame 0x2333ea8c kern_wait6(2730d380,7,0,0,2333eb14,30,0,0) at kern_wait6+0x3b8/frame 0x2333eae8 sys_wait4(2730d380,2730d60c) at sys_wait4+0x79/frame 0x2333ec08 syscall(2333ece8,3b,3b,3b,ffbfe114,...) at syscall+0x306/frame 0x2333ecdc Xint0x80_syscall() at 0xffc033c9/frame 0x2333ecdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command ipcrm pid 70635 tid 100614 td 0x27300e00 (CPU 0) kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command xargs pid 70612 tid 100418 td 0x26bba500 sched_switch(26bba500,104) at sched_switch+0x4f3/frame 0x232fa9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x232fa9f8 sleepq_switch(1680,6c,0,1da1554,18dffaa0,...) at sleepq_switch+0xf7/frame 0x232faa18 sleepq_catch_signals(232faa8c,fdf10e,18dffaa0,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x232faa44 sleepq_wait_sig(18dffaa0,6c) at sleepq_wait_sig+0xe/frame 0x232faa4c _sleep(18dffaa0,18dffb48,36c,1544874,0,0,0,0,100) at _sleep+0x27e/frame 0x232faa8c kern_wait6(26bba500,7,0,0,232fab14,30,0,0) at kern_wait6+0x3b8/frame 0x232faae8 sys_wait4(26bba500,26bba78c) at sys_wait4+0x79/frame 0x232fac08 syscall(232face8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232facdc Xint0x80_syscall() at 0xffc033c9/frame 0x232facdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 70560 tid 100260 td 0x22fb2c00 sched_switch(22fb2c00,104) at sched_switch+0x4f3/frame 0x26f48a50 mi_switch(104) at mi_switch+0x1c4/frame 0x26f48a8c sleepq_switch(2e40,5c,0,1da2d14,27370d50,...) at sleepq_switch+0xf7/frame 0x26f48aac sleepq_catch_signals(26f48b20,fdf10e,1fc211a8,5c,1fc21348,...) at sleepq_catch_signals+0x368/frame 0x26f48ad8 sleepq_wait_sig(1fc211a8,5c) at sleepq_wait_sig+0xe/frame 0x26f48ae0 _sleep(1fc211a8,1fc21338,15c,15288cd,0,0,0,0,100) at _sleep+0x27e/frame 0x26f48b20 pipe_read(18df2038,26f48bd8,18d13e00,0,22fb2c00) at pipe_read+0x3bf/frame 0x26f48b60 dofileread(18df2038,26f48bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x26f48b90 kern_readv(22fb2c00,3,26f48bd8) at kern_readv+0x3b/frame 0x26f48bbc sys_read(22fb2c00,22fb2e8c) at sys_read+0x48/frame 0x26f48c08 syscall(26f48ce8,3b,3b,3b,ffbfe2b8,...) at syscall+0x306/frame 0x26f48cdc Xint0x80_syscall() at 0xffc033c9/frame 0x26f48cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command swap pid 70298 tid 100725 td 0x277f4a80 sched_switch(277f4a80,60b) at sched_switch+0x4f3/frame 0x26f66c70 mi_switch(60b) at mi_switch+0x1c4/frame 0x26f66cac sched_preempt(277f4a80) at sched_preempt+0x7a/frame 0x26f66cc4 ipi_bitmap_handler(3b,3b,3b,0,20a00180,...) at ipi_bitmap_handler+0xa0/frame 0x26f66ce0 Xipi_intr_bitmap_handler() at 0xffc04f87/frame 0x26f66ce0 --- interrupt, eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- exit_tag(0,0,0,0,0,...) at 0xffc01430/frame 0xffbfe750 Tracing command swap pid 70296 tid 100240 td 0x251ffc00 sched_switch(251ffc00,104) at sched_switch+0x4f3/frame 0x26fed9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x26feda08 sleepq_switch(251ffc00,0,1afa800,26feda8c,251ffc00,...) at sleepq_switch+0xf7/frame 0x26feda28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x26feda4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x26feda8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x26fedac0 vm_fault(26b3f2e8,33ce7000,2,0,0) at vm_fault+0x12f9/frame 0x26fedb98 vm_fault_trap(26b3f2e8,33ce7000,2,0,26fedc2c,26fedc30) at vm_fault_trap+0x52/frame 0x26fedbc0 trap_pfault(33ce7000,26fedc2c,26fedc30) at trap_pfault+0x161/frame 0x26fedc14 trap(26fedce8,3b,3b,3b,33ce7000,...) at trap+0x5c8/frame 0x26fedcdc calltrap() at 0xffc0316f/frame 0x26fedcdc --- trap 0, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70295 tid 101346 td 0x2730a880 sched_switch(2730a880,60b) at sched_switch+0x4f3/frame 0x2788bc70 mi_switch(60b) at mi_switch+0x1c4/frame 0x2788bcac sched_preempt(2730a880) at sched_preempt+0x7a/frame 0x2788bcc4 ipi_bitmap_handler(3b,3b,3b,0,20a00180,...) at ipi_bitmap_handler+0xa0/frame 0x2788bce0 Xipi_intr_bitmap_handler() at 0xffc04f87/frame 0x2788bce0 --- interrupt, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70294 tid 101375 td 0x26bb9700 sched_switch(26bb9700,104) at sched_switch+0x4f3/frame 0x27863930 mi_switch(104) at mi_switch+0x1c4/frame 0x27863970 sleepq_switch(26bb9700,0,2614f4fc,278639f4,26bb9700,...) at sleepq_switch+0xf7/frame 0x27863990 sleepq_timedwait(2614f4fc,50,1461fab,9844500,1461fab,...) at sleepq_timedwait+0x3f/frame 0x278639b4 _sleep(2614f4fc,2614f47c,50,147458b,ffffe8e0,13,0,0,100) at _sleep+0x22c/frame 0x278639f4 swap_pager_getpages(2614f47c,27863b00,1,27863b5c,27863b54) at swap_pager_getpages+0x643/frame 0x27863a70 vm_pager_get_pages(2614f47c,27863b00,1,27863b5c,27863b54) at vm_pager_get_pages+0x44/frame 0x27863ab4 vm_fault(9890e88,20429000,1,0,0) at vm_fault+0x7b2/frame 0x27863b98 vm_fault_trap(9890e88,20429090,1,0,27863c2c,27863c30) at vm_fault_trap+0x52/frame 0x27863bc0 trap_pfault(20429090,27863c2c,27863c30) at trap_pfault+0x161/frame 0x27863c14 trap(27863ce8,3b,3b,3b,20800380,...) at trap+0x5c8/frame 0x27863cdc calltrap() at 0xffc0316f/frame 0x27863cdc --- trap 0, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70293 tid 100309 td 0x26bbb700 sched_switch(26bbb700,104) at sched_switch+0x4f3/frame 0x274ce9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x274cea08 sleepq_switch(26bbb700,0,1afa800,274cea8c,26bbb700,...) at sleepq_switch+0xf7/frame 0x274cea28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x274cea4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x274cea8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x274ceac0 vm_fault(26b3f1f0,30aa5000,2,0,0) at vm_fault+0x12f9/frame 0x274ceb98 vm_fault_trap(26b3f1f0,30aa5000,2,0,274cec2c,274cec30) at vm_fault_trap+0x52/frame 0x274cebc0 trap_pfault(30aa5000,274cec2c,274cec30) at trap_pfault+0x161/frame 0x274cec14 trap(274cece8,3b,3b,3b,30aa5000,...) at trap+0x5c8/frame 0x274cecdc calltrap() at 0xffc0316f/frame 0x274cecdc --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70292 tid 100455 td 0x26bb5000 sched_switch(26bb5000,103) at sched_switch+0x4f3/frame 0x26f10938 mi_switch(103,1da46e0) at mi_switch+0x1c4/frame 0x26f10978 turnstile_wait(1fc0ae00,18863180,0) at turnstile_wait+0x2d7/frame 0x26f10994 __rw_wlock_hard(1e202d0,18863184,14af7c6,1289) at __rw_wlock_hard+0x465/frame 0x26f10a24 _rw_wlock_cookie(1e202d0,14af7c6,1289) at _rw_wlock_cookie+0x99/frame 0x26f10a48 pmap_pae_page_is_mapped(43c75e4) at pmap_pae_page_is_mapped+0x24/frame 0x26f10a64 vm_pager_assert_in(1) at vm_pager_assert_in+0x91/frame 0x26f10a80 vm_pager_get_pages(2376d334,26f10b00,1,26f10b5c,26f10b54) at vm_pager_get_pages+0x2a/frame 0x26f10ab4 vm_fault(1fc13000,29f93000,2,0,0) at vm_fault+0x7b2/frame 0x26f10b98 vm_fault_trap(1fc13000,29f93000,2,0,26f10c2c,26f10c30) at vm_fault_trap+0x52/frame 0x26f10bc0 trap_pfault(29f93000,26f10c2c,26f10c30) at trap_pfault+0x161/frame 0x26f10c14 trap(26f10ce8,3b,3b,3b,29f93000,...) at trap+0x5c8/frame 0x26f10cdc calltrap() at 0xffc0316f/frame 0x26f10cdc --- trap 0, eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- exit_tag(0,0,0,0,0,...) at 0xffc01430/frame 0xffbfe750 Tracing command swap pid 70291 tid 100927 td 0x26bba180 sched_switch(26bba180,207) at sched_switch+0x4f3/frame 0x1295dc68 mi_switch(207) at mi_switch+0x1c4/frame 0x1295dca4 ast(1295dce8,3b,3b,3b,0,...) at ast+0x2f3/frame 0x1295dcdc exit_tag(3b,0,40429c,ffbfe750,ffc07fd4,...) at 0xffc055ed/frame 0xffc07fa8 (null)() at 0x3b/frame 0x3b Tracing command swap pid 70290 tid 101073 td 0x26bf9a80 sched_switch(26bf9a80,103) at sched_switch+0x4f3/frame 0x23339938 mi_switch(103,1da46e0) at mi_switch+0x1c4/frame 0x23339978 turnstile_wait(1fc0ae00,18863180,0) at turnstile_wait+0x2d7/frame 0x23339994 __rw_wlock_hard(1e202d0,18863184,14af7c6,1289) at __rw_wlock_hard+0x465/frame 0x23339a24 _rw_wlock_cookie(1e202d0,14af7c6,1289) at _rw_wlock_cookie+0x99/frame 0x23339a48 pmap_pae_page_is_mapped(6255718) at pmap_pae_page_is_mapped+0x24/frame 0x23339a64 vm_pager_assert_in(1) at vm_pager_assert_in+0x91/frame 0x23339a80 vm_pager_get_pages(18124334,23339b00,1,23339b5c,23339b54) at vm_pager_get_pages+0x2a/frame 0x23339ab4 vm_fault(1fc135d0,3181e000,2,0,0) at vm_fault+0x7b2/frame 0x23339b98 vm_fault_trap(1fc135d0,3181e000,2,0,23339c2c,23339c30) at vm_fault_trap+0x52/frame 0x23339bc0 trap_pfault(3181e000,23339c2c,23339c30) at trap_pfault+0x161/frame 0x23339c14 trap(23339ce8,3b,3b,3b,3181e000,...) at trap+0x5c8/frame 0x23339cdc calltrap() at 0xffc0316f/frame 0x23339cdc --- trap 0, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70289 tid 100441 td 0x27288e00 sched_switch(27288e00,104) at sched_switch+0x4f3/frame 0x2061d9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x2061da08 sleepq_switch(27288e00,0,1afa800,2061da8c,27288e00,...) at sleepq_switch+0xf7/frame 0x2061da28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x2061da4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x2061da8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x2061dac0 vm_fault(20a685d0,2d246000,2,0,0) at vm_fault+0x12f9/frame 0x2061db98 vm_fault_trap(20a685d0,2d246000,2,0,2061dc2c,2061dc30) at vm_fault_trap+0x52/frame 0x2061dbc0 trap_pfault(2d246000,2061dc2c,2061dc30) at trap_pfault+0x161/frame 0x2061dc14 trap(2061dce8,3b,3b,3b,2d246000,...) at trap+0x5c8/frame 0x2061dcdc calltrap() at 0xffc0316f/frame 0x2061dcdc --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70288 tid 100218 td 0x22fb5700 (CPU 1) cpustop_handler(200046,ffc0afa8,ffc0af9c,1417ccb,0,...) at cpustop_handler+0x24/frame 0xffc0aed0 ipi_nmi_handler(0,0,0,0,26bc5aa0,...) at ipi_nmi_handler+0x31/frame 0xffc0aee0 trap(ffc0afa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0af9c calltrap() at 0xffc0316f/frame 0xffc0af9c --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70287 tid 100241 td 0x26b14700 sched_switch(26b14700,104) at sched_switch+0x4f3/frame 0x2062f9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x2062fa08 sleepq_switch(26b14700,0,1afa800,2062fa8c,26b14700,...) at sleepq_switch+0xf7/frame 0x2062fa28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x2062fa4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x2062fa8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x2062fac0 vm_fault(26b41ba0,2ca1c000,2,0,0) at vm_fault+0x12f9/frame 0x2062fb98 vm_fault_trap(26b41ba0,2ca1c000,2,0,2062fc2c,2062fc30) at vm_fault_trap+0x52/frame 0x2062fbc0 trap_pfault(2ca1c000,2062fc2c,2062fc30) at trap_pfault+0x161/frame 0x2062fc14 trap(2062fce8,3b,3b,3b,2ca1c000,...) at trap+0x5c8/frame 0x2062fcdc calltrap() at 0xffc0316f/frame 0x2062fcdc --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70286 tid 101028 td 0x27289880 (CPU 3) cpustop_handler(200046,ffc0dfa8,ffc0df9c,1417ccb,0,...) at cpustop_handler+0x24/frame 0xffc0ded0 ipi_nmi_handler(0,0,0,0,237a4d50,...) at ipi_nmi_handler+0x31/frame 0xffc0dee0 trap(ffc0dfa8,3b,3b,3b,0,...) at trap+0xbb/frame 0xffc0df9c calltrap() at 0xffc0316f/frame 0xffc0df9c --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70285 tid 100258 td 0x277f5880 sched_switch(277f5880,104) at sched_switch+0x4f3/frame 0x275919c8 mi_switch(104) at mi_switch+0x1c4/frame 0x27591a08 sleepq_switch(277f5880,0,1afa800,27591a8c,277f5880,...) at sleepq_switch+0xf7/frame 0x27591a28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x27591a4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x27591a8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x27591ac0 vm_fault(1fc111f0,2efee000,2,0,0) at vm_fault+0x12f9/frame 0x27591b98 vm_fault_trap(1fc111f0,2efee180,2,0,27591c2c,27591c30) at vm_fault_trap+0x52/frame 0x27591bc0 trap_pfault(2efee180,27591c2c,27591c30) at trap_pfault+0x161/frame 0x27591c14 trap(27591ce8,3b,3b,3b,0,...) at trap+0x5c8/frame 0x27591cdc calltrap() at 0xffc0316f/frame 0x27591cdc --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70284 tid 100189 td 0x201b9380 sched_switch(201b9380,104) at sched_switch+0x4f3/frame 0x275f09c8 mi_switch(104) at mi_switch+0x1c4/frame 0x275f0a08 sleepq_switch(201b9380,0,1afa800,275f0a8c,201b9380,...) at sleepq_switch+0xf7/frame 0x275f0a28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x275f0a4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x275f0a8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x275f0ac0 vm_fault(1fc118b8,29ff8000,2,0,0) at vm_fault+0x12f9/frame 0x275f0b98 vm_fault_trap(1fc118b8,29ff8180,2,0,275f0c2c,275f0c30) at vm_fault_trap+0x52/frame 0x275f0bc0 trap_pfault(29ff8180,275f0c2c,275f0c30) at trap_pfault+0x161/frame 0x275f0c14 trap(275f0ce8,3b,3b,3b,0,...) at trap+0x5c8/frame 0x275f0cdc calltrap() at 0xffc0316f/frame 0x275f0cdc --- trap 0, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70282 tid 100643 td 0x1fc0fc00 sched_switch(1fc0fc00,104) at sched_switch+0x4f3/frame 0x2752c9c8 mi_switch(104) at mi_switch+0x1c4/frame 0x2752ca08 sleepq_switch(1fc0fc00,0,1afa800,2752ca8c,1fc0fc00,...) at sleepq_switch+0xf7/frame 0x2752ca28 sleepq_timedwait(1afa800,78,40000,1,155a962,...) at sleepq_timedwait+0x3f/frame 0x2752ca4c _sleep(1afa800,1af9fc0,278,143b1de,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x2752ca8c vm_waitpfault(1d8cf48,2710) at vm_waitpfault+0x7a/frame 0x2752cac0 vm_fault(26b410f8,30247000,2,0,0) at vm_fault+0x12f9/frame 0x2752cb98 vm_fault_trap(26b410f8,30247180,2,0,2752cc2c,2752cc30) at vm_fault_trap+0x52/frame 0x2752cbc0 trap_pfault(30247180,2752cc2c,2752cc30) at trap_pfault+0x161/frame 0x2752cc14 trap(2752cce8,3b,3b,3b,0,...) at trap+0x5c8/frame 0x2752ccdc calltrap() at 0xffc0316f/frame 0x2752ccdc --- trap 0x13, eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command swap pid 70281 tid 100239 td 0x2730a180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(38,a07e50,38,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(2730a180,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command swap pid 70273 tid 100959 td 0x23177880 sched_switch(23177880,104) at sched_switch+0x4f3/frame 0x26f61a68 mi_switch(104) at mi_switch+0x1c4/frame 0x26f61aa4 sleepq_switch(37c0,6c,0,1da3694,237a574c,...) at sleepq_switch+0xf7/frame 0x26f61ac4 sleepq_catch_signals(0,152873e,23177880,100,1d921fe,...) at sleepq_catch_signals+0x368/frame 0x26f61af0 sleepq_timedwait_sig(1d921fe,6c,14e6927,60a,26f61b98,...) at sleepq_timedwait_sig+0x17/frame 0x26f61b1c _sleep(1d921fe,0,16c,153b48a,b05735ae,4b0,10000000,0,200) at _sleep+0x21c/frame 0x26f61b5c kern_clock_nanosleep(23177880,0,0,26f61be8,26f61be0) at kern_clock_nanosleep+0x1b4/frame 0x26f61bc4 sys_nanosleep(23177880,23177b0c) at sys_nanosleep+0x55/frame 0x26f61c08 syscall(26f61ce8,3b,3b,3b,1,...) at syscall+0x306/frame 0x26f61cdc Xint0x80_syscall() at 0xffc033c9/frame 0x26f61cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command md10 pid 70267 tid 101350 td 0x26bb9380 sched_switch(26bb9380,104) at sched_switch+0x4f3/frame 0x27518bd0 mi_switch(104) at mi_switch+0x1c4/frame 0x27518c0c sleepq_switch(23045000,5c) at sleepq_switch+0xf7/frame 0x27518c2c _sleep(23045000,23045028,25c,14eacc0,0,0,0,0,100) at _sleep+0x2f7/frame 0x27518c6c md_kthread(23045000,27518ce8) at md_kthread+0xc2/frame 0x27518cb4 fork_exit(be0810,23045000,27518ce8,0,0,...) at fork_exit+0x69/frame 0x27518cd4 fork_trampoline() at 0xffc033de/frame 0x27518cd4 --- kthread start Tracing command sh pid 70226 tid 100577 td 0x22fb6c00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(36,a07e50,36,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(22fb6c00,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command tee pid 70225 tid 100323 td 0x2728aa80 sched_switch(2728aa80,104) at sched_switch+0x4f3/frame 0x27827a50 mi_switch(104) at mi_switch+0x1c4/frame 0x27827a8c sleepq_switch(300,5c,0,1da01d4,26b1374c,...) at sleepq_switch+0xf7/frame 0x27827aac sleepq_catch_signals(27827b20,fdf10e,2048f4f8,5c,2048f698,...) at sleepq_catch_signals+0x368/frame 0x27827ad8 sleepq_wait_sig(2048f4f8,5c) at sleepq_wait_sig+0xe/frame 0x27827ae0 _sleep(2048f4f8,2048f688,15c,15288cd,0,0,0,0,100) at _sleep+0x27e/frame 0x27827b20 pipe_read(1fda6690,27827bd8,9d1f800,0,2728aa80) at pipe_read+0x3bf/frame 0x27827b60 dofileread(1fda6690,27827bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x27827b90 kern_readv(2728aa80,0,27827bd8) at kern_readv+0x3b/frame 0x27827bbc sys_read(2728aa80,2728ad0c) at sys_read+0x48/frame 0x27827c08 syscall(27827ce8,3b,3b,3b,ffbfe7f0,...) at syscall+0x306/frame 0x27827cdc Xint0x80_syscall() at 0xffc033c9/frame 0x27827cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 70224 tid 100391 td 0x27289c00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(36,a07e50,36,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(27289c00,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command aiod4 pid 2313 tid 100109 td 0x181cdc00 sched_switch(181cdc00,104) at sched_switch+0x4f3/frame 0x1fac6ba8 mi_switch(104) at mi_switch+0x1c4/frame 0x1fac6be8 sleepq_switch(181cdc00,0,1885a9fc,1fac6c6c,181cdc00,...) at sleepq_switch+0xf7/frame 0x1fac6c08 sleepq_timedwait(1885a9fc,5c,10000,10000,150c3b2,...) at sleepq_timedwait+0x3f/frame 0x1fac6c2c _sleep(1885a9fc,1dfea60,5c,148f926,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x1fac6c6c aio_daemon(4,1fac6ce8) at aio_daemon+0x282/frame 0x1fac6cb4 fork_exit(1074290,4,1fac6ce8,0,0,...) at fork_exit+0x69/frame 0x1fac6cd4 fork_trampoline() at 0xffc033de/frame 0x1fac6cd4 --- kthread start Tracing command aiod3 pid 2312 tid 100209 td 0x22fb2180 sched_switch(22fb2180,104) at sched_switch+0x4f3/frame 0x23325ba8 mi_switch(104) at mi_switch+0x1c4/frame 0x23325be8 sleepq_switch(22fb2180,0,26b129fc,23325c6c,22fb2180,...) at sleepq_switch+0xf7/frame 0x23325c08 sleepq_timedwait(26b129fc,5c,10000,10000,150c3b2,...) at sleepq_timedwait+0x3f/frame 0x23325c2c _sleep(26b129fc,1dfea60,5c,148f926,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x23325c6c aio_daemon(3,23325ce8) at aio_daemon+0x282/frame 0x23325cb4 fork_exit(1074290,3,23325ce8,0,0,...) at fork_exit+0x69/frame 0x23325cd4 fork_trampoline() at 0xffc033de/frame 0x23325cd4 --- kthread start Tracing command aiod2 pid 2311 tid 100179 td 0x23176e00 sched_switch(23176e00,104) at sched_switch+0x4f3/frame 0x2323cba8 mi_switch(104) at mi_switch+0x1c4/frame 0x2323cbe8 sleepq_switch(23176e00,0,18da26a8,2323cc6c,23176e00,...) at sleepq_switch+0xf7/frame 0x2323cc08 sleepq_timedwait(18da26a8,5c,10000,10000,150c3b2,...) at sleepq_timedwait+0x3f/frame 0x2323cc2c _sleep(18da26a8,1dfea60,5c,148f926,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x2323cc6c aio_daemon(2,2323cce8) at aio_daemon+0x282/frame 0x2323ccb4 fork_exit(1074290,2,2323cce8,0,0,...) at fork_exit+0x69/frame 0x2323ccd4 fork_trampoline() at 0xffc033de/frame 0x2323ccd4 --- kthread start Tracing command aiod1 pid 2310 tid 100219 td 0x1fc0f880 sched_switch(1fc0f880,104) at sched_switch+0x4f3/frame 0x26e8aba8 mi_switch(104) at mi_switch+0x1c4/frame 0x26e8abe8 sleepq_switch(1fc0f880,0,1886174c,26e8ac6c,1fc0f880,...) at sleepq_switch+0xf7/frame 0x26e8ac08 sleepq_timedwait(1886174c,5c,10000,10000,150c3b2,...) at sleepq_timedwait+0x3f/frame 0x26e8ac2c _sleep(1886174c,1dfea60,5c,148f926,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0x26e8ac6c aio_daemon(1,26e8ace8) at aio_daemon+0x282/frame 0x26e8acb4 fork_exit(1074290,1,26e8ace8,0,0,...) at fork_exit+0x69/frame 0x26e8acd4 fork_trampoline() at 0xffc033de/frame 0x26e8acd4 --- kthread start Tracing command sh pid 1069 tid 100111 td 0x181cd880 sched_switch(181cd880,104) at sched_switch+0x4f3/frame 0x1faf8a50 mi_switch(104) at mi_switch+0x1c4/frame 0x1faf8a8c sleepq_switch(1000,5c,0,1da0ed4,1885a6a8,...) at sleepq_switch+0xf7/frame 0x1faf8aac sleepq_catch_signals(1faf8b20,fdf10e,20314000,5c,203141a0,...) at sleepq_catch_signals+0x368/frame 0x1faf8ad8 sleepq_wait_sig(20314000,5c) at sleepq_wait_sig+0xe/frame 0x1faf8ae0 _sleep(20314000,20314190,15c,15288cd,0,0,0,0,100) at _sleep+0x27e/frame 0x1faf8b20 pipe_read(18df09d8,1faf8bd8,18d13e00,0,181cd880) at pipe_read+0x3bf/frame 0x1faf8b60 dofileread(18df09d8,1faf8bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1faf8b90 kern_readv(181cd880,0,1faf8bd8) at kern_readv+0x3b/frame 0x1faf8bbc sys_read(181cd880,181cdb0c) at sys_read+0x48/frame 0x1faf8c08 syscall(1faf8ce8,3b,3b,3b,206f6584,...) at syscall+0x306/frame 0x1faf8cdc Xint0x80_syscall() at 0xffc033c9/frame 0x1faf8cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command awk pid 1068 tid 100175 td 0x22fb3a80 sched_switch(22fb3a80,104) at sched_switch+0x4f3/frame 0x23208a50 mi_switch(104) at mi_switch+0x1c4/frame 0x23208a8c sleepq_switch(3740,5c,0,1da3614,18183354,...) at sleepq_switch+0xf7/frame 0x23208aac sleepq_catch_signals(23208b20,fdf10e,1fc59d40,5c,1fc59ee0,...) at sleepq_catch_signals+0x368/frame 0x23208ad8 sleepq_wait_sig(1fc59d40,5c) at sleepq_wait_sig+0xe/frame 0x23208ae0 _sleep(1fc59d40,1fc59ed0,15c,15288cd,0,0,0,0,100) at _sleep+0x27e/frame 0x23208b20 pipe_read(18daf930,23208bd8,18d13e00,0,22fb3a80) at pipe_read+0x3bf/frame 0x23208b60 dofileread(18daf930,23208bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x23208b90 kern_readv(22fb3a80,0,23208bd8) at kern_readv+0x3b/frame 0x23208bbc sys_read(22fb3a80,22fb3d0c) at sys_read+0x48/frame 0x23208c08 syscall(23208ce8,3b,3b,3b,2064f5cc,...) at syscall+0x306/frame 0x23208cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23208cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sh pid 1067 tid 100210 td 0x22fb1e00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,35,a07e50,35,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(35,a07e50,35,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(22fb1e00,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sh pid 1046 tid 100182 td 0x1fdb5a80 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,35,a07e50,35,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(35,a07e50,35,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(1fdb5a80,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sh pid 949 tid 100197 td 0x201b7380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,34,a07e50,34,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(34,a07e50,34,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(201b7380,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command bash pid 913 tid 100199 td 0x25201180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(36,a07e50,36,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(25201180,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command csh pid 911 tid 100203 td 0x1fdb5380 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,35,a07e50,35,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(35,a07e50,35,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(1fdb5380,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command su pid 910 tid 100202 td 0x25200700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,34,a07e50,34,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(34,a07e50,34,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(25200700,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command bash pid 901 tid 100204 td 0x1fdb5000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(36,a07e50,36,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(1fdb5000,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sshd pid 900 tid 100200 td 0x25200e00 sched_switch(25200e00,104) at sched_switch+0x4f3/frame 0x232f5970 mi_switch(104) at mi_switch+0x1c4/frame 0x232f59ac sleepq_switch(680,0,25200e00,1da0554,237a26a8,...) at sleepq_switch+0xf7/frame 0x232f59cc sleepq_catch_signals(232f5a38,f76998,17edfee4,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x232f59f8 sleepq_wait_sig(17edfee4,0) at sleepq_wait_sig+0xe/frame 0x232f5a00 _cv_wait_sig(17edfee4,17edfed0) at _cv_wait_sig+0x138/frame 0x232f5a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x232f5a54 kern_select(25200e00,e,2047a270,2047a278,0,0,20) at kern_select+0x8ed/frame 0x232f5bd4 sys_select(25200e00,2520108c) at sys_select+0x41/frame 0x232f5c08 syscall(232f5ce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x232f5cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232f5cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command sshd pid 898 tid 100198 td 0x201b7000 sched_switch(201b7000,104) at sched_switch+0x4f3/frame 0x232cb998 mi_switch(104) at mi_switch+0x1c4/frame 0x232cb9d4 sleepq_switch(1b80,0,201b7000,1da1a54,237a2d50,...) at sleepq_switch+0xf7/frame 0x232cb9f4 sleepq_catch_signals(232cba60,f76998,9df4a24,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x232cba20 sleepq_wait_sig(9df4a24,0) at sleepq_wait_sig+0xe/frame 0x232cba28 _cv_wait_sig(9df4a24,9df4a10) at _cv_wait_sig+0x138/frame 0x232cba60 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x232cba7c kern_poll(201b7000,ffbfe800,1,0,0) at kern_poll+0x476/frame 0x232cbbd4 sys_poll(201b7000,201b728c) at sys_poll+0x50/frame 0x232cbc08 syscall(232cbce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x232cbcdc Xint0x80_syscall() at 0xffc033c9/frame 0x232cbcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 770 tid 100196 td 0x201b7700 sched_switch(201b7700,104) at sched_switch+0x4f3/frame 0x232c1a00 mi_switch(104) at mi_switch+0x1c4/frame 0x232c1a3c sleepq_switch(1600,0,201b7700,1da14d4,237a33f8,...) at sleepq_switch+0xf7/frame 0x232c1a5c sleepq_catch_signals(232c1ac8,f76998,18202c74,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232c1a88 sleepq_wait_sig(18202c74,0) at sleepq_wait_sig+0xe/frame 0x232c1a90 _cv_wait_sig(18202c74,18202c04) at _cv_wait_sig+0x138/frame 0x232c1ac8 tty_wait(18202c00,18202c74) at tty_wait+0x35/frame 0x232c1ae0 ttydisc_read(18202c00,232c1bd8,0) at ttydisc_read+0x1c6/frame 0x232c1b14 ttydev_read(9966000,232c1bd8,0) at ttydev_read+0x3d/frame 0x232c1b30 devfs_read_f(18df10a8,232c1bd8,237a9500,0,201b7700) at devfs_read_f+0xaa/frame 0x232c1b60 dofileread(18df10a8,232c1bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232c1b90 kern_readv(201b7700,0,232c1bd8) at kern_readv+0x3b/frame 0x232c1bbc sys_read(201b7700,201b798c) at sys_read+0x48/frame 0x232c1c08 syscall(232c1ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232c1cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232c1cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command getty pid 769 tid 100195 td 0x201b7a80 sched_switch(201b7a80,104) at sched_switch+0x4f3/frame 0x232bca00 mi_switch(104) at mi_switch+0x1c4/frame 0x232bca3c sleepq_switch(2800,0,201b7a80,1da26d4,237a374c,...) at sleepq_switch+0xf7/frame 0x232bca5c sleepq_catch_signals(232bcac8,f76998,1887d474,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232bca88 sleepq_wait_sig(1887d474,0) at sleepq_wait_sig+0xe/frame 0x232bca90 _cv_wait_sig(1887d474,1887d404) at _cv_wait_sig+0x138/frame 0x232bcac8 tty_wait(1887d400,1887d474) at tty_wait+0x35/frame 0x232bcae0 ttydisc_read(1887d400,232bcbd8,0) at ttydisc_read+0x1c6/frame 0x232bcb14 ttydev_read(18365000,232bcbd8,0) at ttydev_read+0x3d/frame 0x232bcb30 devfs_read_f(18dafa10,232bcbd8,18d14500,0,201b7a80) at devfs_read_f+0xaa/frame 0x232bcb60 dofileread(18dafa10,232bcbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232bcb90 kern_readv(201b7a80,0,232bcbd8) at kern_readv+0x3b/frame 0x232bcbbc sys_read(201b7a80,201b7d0c) at sys_read+0x48/frame 0x232bcc08 syscall(232bcce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232bccdc Xint0x80_syscall() at 0xffc033c9/frame 0x232bccdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command getty pid 768 tid 100194 td 0x201b7e00 sched_switch(201b7e00,104) at sched_switch+0x4f3/frame 0x232b7a00 mi_switch(104) at mi_switch+0x1c4/frame 0x232b7a3c sleepq_switch(2b00,0,201b7e00,1da29d4,237a3aa0,...) at sleepq_switch+0xf7/frame 0x232b7a5c sleepq_catch_signals(232b7ac8,f76998,1887d874,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232b7a88 sleepq_wait_sig(1887d874,0) at sleepq_wait_sig+0xe/frame 0x232b7a90 _cv_wait_sig(1887d874,1887d804) at _cv_wait_sig+0x138/frame 0x232b7ac8 tty_wait(1887d800,1887d874) at tty_wait+0x35/frame 0x232b7ae0 ttydisc_read(1887d800,232b7bd8,0) at ttydisc_read+0x1c6/frame 0x232b7b14 ttydev_read(18365200,232b7bd8,0) at ttydev_read+0x3d/frame 0x232b7b30 devfs_read_f(18db07a8,232b7bd8,237e5a00,0,201b7e00) at devfs_read_f+0xaa/frame 0x232b7b60 dofileread(18db07a8,232b7bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232b7b90 kern_readv(201b7e00,0,232b7bd8) at kern_readv+0x3b/frame 0x232b7bbc sys_read(201b7e00,201b808c) at sys_read+0x48/frame 0x232b7c08 syscall(232b7ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232b7cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232b7cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command getty pid 767 tid 100193 td 0x201b8180 sched_switch(201b8180,104) at sched_switch+0x4f3/frame 0x232b2a00 mi_switch(104) at mi_switch+0x1c4/frame 0x232b2a3c sleepq_switch(2a00,0,201b8180,1da28d4,237a4000,...) at sleepq_switch+0xf7/frame 0x232b2a5c sleepq_catch_signals(232b2ac8,f76998,1887dc74,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232b2a88 sleepq_wait_sig(1887dc74,0) at sleepq_wait_sig+0xe/frame 0x232b2a90 _cv_wait_sig(1887dc74,1887dc04) at _cv_wait_sig+0x138/frame 0x232b2ac8 tty_wait(1887dc00,1887dc74) at tty_wait+0x35/frame 0x232b2ae0 ttydisc_read(1887dc00,232b2bd8,0) at ttydisc_read+0x1c6/frame 0x232b2b14 ttydev_read(18365400,232b2bd8,0) at ttydev_read+0x3d/frame 0x232b2b30 devfs_read_f(18daf9d8,232b2bd8,9d23000,0,201b8180) at devfs_read_f+0xaa/frame 0x232b2b60 dofileread(18daf9d8,232b2bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232b2b90 kern_readv(201b8180,0,232b2bd8) at kern_readv+0x3b/frame 0x232b2bbc sys_read(201b8180,201b840c) at sys_read+0x48/frame 0x232b2c08 syscall(232b2ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232b2cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232b2cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 766 tid 100192 td 0x201b8500 sched_switch(201b8500,104) at sched_switch+0x4f3/frame 0x232ada00 mi_switch(104) at mi_switch+0x1c4/frame 0x232ada3c sleepq_switch(2500,0,201b8500,1da23d4,237a4354,...) at sleepq_switch+0xf7/frame 0x232ada5c sleepq_catch_signals(232adac8,f76998,1887e074,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232ada88 sleepq_wait_sig(1887e074,0) at sleepq_wait_sig+0xe/frame 0x232ada90 _cv_wait_sig(1887e074,1887e004) at _cv_wait_sig+0x138/frame 0x232adac8 tty_wait(1887e000,1887e074) at tty_wait+0x35/frame 0x232adae0 ttydisc_read(1887e000,232adbd8,0) at ttydisc_read+0x1c6/frame 0x232adb14 ttydev_read(18365600,232adbd8,0) at ttydev_read+0x3d/frame 0x232adb30 devfs_read_f(18df1118,232adbd8,237e5b00,0,201b8500) at devfs_read_f+0xaa/frame 0x232adb60 dofileread(18df1118,232adbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232adb90 kern_readv(201b8500,0,232adbd8) at kern_readv+0x3b/frame 0x232adbbc sys_read(201b8500,201b878c) at sys_read+0x48/frame 0x232adc08 syscall(232adce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232adcdc Xint0x80_syscall() at 0xffc033c9/frame 0x232adcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 765 tid 100191 td 0x201b8880 sched_switch(201b8880,104) at sched_switch+0x4f3/frame 0x232a8a00 mi_switch(104) at mi_switch+0x1c4/frame 0x232a8a3c sleepq_switch(2400,0,201b8880,1da22d4,237a46a8,...) at sleepq_switch+0xf7/frame 0x232a8a5c sleepq_catch_signals(232a8ac8,f76998,1887e474,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232a8a88 sleepq_wait_sig(1887e474,0) at sleepq_wait_sig+0xe/frame 0x232a8a90 _cv_wait_sig(1887e474,1887e404) at _cv_wait_sig+0x138/frame 0x232a8ac8 tty_wait(1887e400,1887e474) at tty_wait+0x35/frame 0x232a8ae0 ttydisc_read(1887e400,232a8bd8,0) at ttydisc_read+0x1c6/frame 0x232a8b14 ttydev_read(18365800,232a8bd8,0) at ttydev_read+0x3d/frame 0x232a8b30 devfs_read_f(18df1268,232a8bd8,237a9600,0,201b8880) at devfs_read_f+0xaa/frame 0x232a8b60 dofileread(18df1268,232a8bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232a8b90 kern_readv(201b8880,0,232a8bd8) at kern_readv+0x3b/frame 0x232a8bbc sys_read(201b8880,201b8b0c) at sys_read+0x48/frame 0x232a8c08 syscall(232a8ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232a8cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232a8cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command getty pid 764 tid 100190 td 0x201b8c00 sched_switch(201b8c00,104) at sched_switch+0x4f3/frame 0x232a3a00 mi_switch(104) at mi_switch+0x1c4/frame 0x232a3a3c sleepq_switch(2700,0,201b8c00,1da25d4,237a49fc,...) at sleepq_switch+0xf7/frame 0x232a3a5c sleepq_catch_signals(232a3ac8,f76998,1887e874,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x232a3a88 sleepq_wait_sig(1887e874,0) at sleepq_wait_sig+0xe/frame 0x232a3a90 _cv_wait_sig(1887e874,1887e804) at _cv_wait_sig+0x138/frame 0x232a3ac8 tty_wait(1887e800,1887e874) at tty_wait+0x35/frame 0x232a3ae0 ttydisc_read(1887e800,232a3bd8,0) at ttydisc_read+0x1c6/frame 0x232a3b14 ttydev_read(18365a00,232a3bd8,0) at ttydev_read+0x3d/frame 0x232a3b30 devfs_read_f(18dafa48,232a3bd8,18d14400,0,201b8c00) at devfs_read_f+0xaa/frame 0x232a3b60 dofileread(18dafa48,232a3bd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x232a3b90 kern_readv(201b8c00,0,232a3bd8) at kern_readv+0x3b/frame 0x232a3bbc sys_read(201b8c00,201b8e8c) at sys_read+0x48/frame 0x232a3c08 syscall(232a3ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x232a3cdc Xint0x80_syscall() at 0xffc033c9/frame 0x232a3cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 763 tid 100122 td 0x201b9e00 sched_switch(201b9e00,104) at sched_switch+0x4f3/frame 0x1fb8fa00 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb8fa3c sleepq_switch(2600,0,201b9e00,1da24d4,1818a74c,...) at sleepq_switch+0xf7/frame 0x1fb8fa5c sleepq_catch_signals(1fb8fac8,f76998,1887ec74,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x1fb8fa88 sleepq_wait_sig(1887ec74,0) at sleepq_wait_sig+0xe/frame 0x1fb8fa90 _cv_wait_sig(1887ec74,1887ec04) at _cv_wait_sig+0x138/frame 0x1fb8fac8 tty_wait(1887ec00,1887ec74) at tty_wait+0x35/frame 0x1fb8fae0 ttydisc_read(1887ec00,1fb8fbd8,0) at ttydisc_read+0x1c6/frame 0x1fb8fb14 ttydev_read(18365c00,1fb8fbd8,0) at ttydev_read+0x3d/frame 0x1fb8fb30 devfs_read_f(18dafa80,1fb8fbd8,18d14600,0,201b9e00) at devfs_read_f+0xaa/frame 0x1fb8fb60 dofileread(18dafa80,1fb8fbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1fb8fb90 kern_readv(201b9e00,0,1fb8fbd8) at kern_readv+0x3b/frame 0x1fb8fbbc sys_read(201b9e00,201ba08c) at sys_read+0x48/frame 0x1fb8fc08 syscall(1fb8fce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x1fb8fcdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fb8fcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command getty pid 762 tid 100112 td 0x181cd500 sched_switch(181cd500,104) at sched_switch+0x4f3/frame 0x1fafda00 mi_switch(104) at mi_switch+0x1c4/frame 0x1fafda3c sleepq_switch(2100,0,181cd500,1da1fd4,1885a354,...) at sleepq_switch+0xf7/frame 0x1fafda5c sleepq_catch_signals(1fafdac8,f76998,1887f074,0,154f261,...) at sleepq_catch_signals+0x368/frame 0x1fafda88 sleepq_wait_sig(1887f074,0) at sleepq_wait_sig+0xe/frame 0x1fafda90 _cv_wait_sig(1887f074,1887f004) at _cv_wait_sig+0x138/frame 0x1fafdac8 tty_wait(1887f000,1887f074) at tty_wait+0x35/frame 0x1fafdae0 ttydisc_read(1887f000,1fafdbd8,0) at ttydisc_read+0x1c6/frame 0x1fafdb14 ttydev_read(18365e00,1fafdbd8,0) at ttydev_read+0x3d/frame 0x1fafdb30 devfs_read_f(18df10e0,1fafdbd8,9d23100,0,181cd500) at devfs_read_f+0xaa/frame 0x1fafdb60 dofileread(18df10e0,1fafdbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x1fafdb90 kern_readv(181cd500,0,1fafdbd8) at kern_readv+0x3b/frame 0x1fafdbbc sys_read(181cd500,181cd78c) at sys_read+0x48/frame 0x1fafdc08 syscall(1fafdce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x1fafdcdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fafdcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command inetd pid 745 tid 100183 td 0x1fdb5700 sched_switch(1fdb5700,104) at sched_switch+0x4f3/frame 0x23250970 mi_switch(104) at mi_switch+0x1c4/frame 0x232509ac sleepq_switch(400,0,1fdb5700,1da02d4,18da06a8,...) at sleepq_switch+0xf7/frame 0x232509cc sleepq_catch_signals(23250a38,f76998,1837b4a4,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x232509f8 sleepq_wait_sig(1837b4a4,0) at sleepq_wait_sig+0xe/frame 0x23250a00 _cv_wait_sig(1837b4a4,1837b490) at _cv_wait_sig+0x138/frame 0x23250a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x23250a54 kern_select(1fdb5700,9,ffbfe638,0,0,0,20) at kern_select+0x8ed/frame 0x23250bd4 sys_select(1fdb5700,1fdb598c) at sys_select+0x41/frame 0x23250c08 syscall(23250ce8,3b,3b,3b,ffbfe6b8,...) at syscall+0x306/frame 0x23250cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23250cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfscbd pid 708 tid 100121 td 0x201ba180 sched_switch(201ba180,104) at sched_switch+0x4f3/frame 0x1fb4a888 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb4a8c4 sleepq_switch(1c0,0,1dceb20,1da0094,1818aaa0,...) at sleepq_switch+0xf7/frame 0x1fb4a8e4 sleepq_catch_signals(e2,201ba180,1abe268,201ba180,1fb4a944,...) at sleepq_catch_signals+0x368/frame 0x1fb4a910 sleepq_timedwait_sig(20545750,0) at sleepq_timedwait_sig+0x17/frame 0x1fb4a940 _cv_timedwait_sig_sbt(20545750,186520c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fb4a97c svc_run_internal(22fb3000,186520c0,186520d0,1865211c,0,...) at svc_run_internal+0x97d/frame 0x1fb4aa60 svc_run(18652000) at svc_run+0x192/frame 0x1fb4aa90 nfscbd_nfsd(201ba180,1fb4ab48) at nfscbd_nfsd+0xd0/frame 0x1fb4ab20 nfssvc_nfscl(201ba180,201ba40c) at nfssvc_nfscl+0x243/frame 0x1fb4abf0 sys_nfssvc(201ba180,201ba40c) at sys_nfssvc+0x92/frame 0x1fb4ac08 syscall(1fb4ace8,3b,3b,3b,1,...) at syscall+0x306/frame 0x1fb4acdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fb4acdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfscbd pid 708 tid 100184 td 0x22fb3700 sched_switch(22fb3700,104) at sched_switch+0x4f3/frame 0x23259b18 mi_switch(104) at mi_switch+0x1c4/frame 0x23259b54 sleepq_switch(2e40,0,22fb3700,1da2d14,1818aaa0,...) at sleepq_switch+0xf7/frame 0x23259b74 sleepq_catch_signals(23259be0,f76998,18d0e950,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x23259ba0 sleepq_wait_sig(18d0e950,0) at sleepq_wait_sig+0xe/frame 0x23259ba8 _cv_wait_sig(18d0e950,186520c0) at _cv_wait_sig+0x138/frame 0x23259be0 svc_run_internal(23259cd4,f98ac9,186520c0,23259ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x23259cac svc_thread_start(186520c0,23259ce8) at svc_thread_start+0xd/frame 0x23259cb4 fork_exit(12559b0,186520c0,23259ce8,3b,3b,...) at fork_exit+0x69/frame 0x23259cd4 fork_trampoline() at 0xffc033de/frame 0x23259cd4 --- kthread start Tracing command nfscbd pid 708 tid 100185 td 0x22fb3380 sched_switch(22fb3380,104) at sched_switch+0x4f3/frame 0x2325eb18 mi_switch(104) at mi_switch+0x1c4/frame 0x2325eb54 sleepq_switch(e00,0,22fb3380,1da0cd4,1818aaa0,...) at sleepq_switch+0xf7/frame 0x2325eb74 sleepq_catch_signals(2325ebe0,f76998,18d0e8d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2325eba0 sleepq_wait_sig(18d0e8d0,0) at sleepq_wait_sig+0xe/frame 0x2325eba8 _cv_wait_sig(18d0e8d0,186520c0) at _cv_wait_sig+0x138/frame 0x2325ebe0 svc_run_internal(2325ecd4,f98ac9,186520c0,2325ece8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2325ecac svc_thread_start(186520c0,2325ece8) at svc_thread_start+0xd/frame 0x2325ecb4 fork_exit(12559b0,186520c0,2325ece8,3b,3b,...) at fork_exit+0x69/frame 0x2325ecd4 fork_trampoline() at 0xffc033de/frame 0x2325ecd4 --- kthread start Tracing command nfscbd pid 708 tid 100186 td 0x22fb3000 sched_switch(22fb3000,104) at sched_switch+0x4f3/frame 0x23263b18 mi_switch(104) at mi_switch+0x1c4/frame 0x23263b54 sleepq_switch(2e00,0,22fb3000,1da2cd4,1818aaa0,...) at sleepq_switch+0xf7/frame 0x23263b74 sleepq_catch_signals(23263be0,f76998,18d0e850,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x23263ba0 sleepq_wait_sig(18d0e850,0) at sleepq_wait_sig+0xe/frame 0x23263ba8 _cv_wait_sig(18d0e850,186520c0) at _cv_wait_sig+0x138/frame 0x23263be0 svc_run_internal(23263cd4,f98ac9,186520c0,23263ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x23263cac svc_thread_start(186520c0,23263ce8) at svc_thread_start+0xd/frame 0x23263cb4 fork_exit(12559b0,186520c0,23263ce8,3b,3b,...) at fork_exit+0x69/frame 0x23263cd4 fork_trampoline() at 0xffc033de/frame 0x23263cd4 --- kthread start Tracing command nfscbd pid 707 tid 100181 td 0x1fdb5e00 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(38,a07e50,38,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(1fdb5e00,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command cron pid 674 tid 100171 td 0x18871180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,36,a07e50,36,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(36,a07e50,36,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(18871180,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sendmail pid 670 tid 100176 td 0x1fdb6180 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,3a,a07e50,3a,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(3a,a07e50,3a,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(1fdb6180,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sendmail pid 667 tid 100177 td 0x23177500 sched_switch(23177500,104) at sched_switch+0x4f3/frame 0x23226930 mi_switch(104) at mi_switch+0x1c4/frame 0x2322696c sleepq_switch(3980,0,1dcec80,1da3854,18da2d50,...) at sleepq_switch+0xf7/frame 0x2322698c sleepq_catch_signals(e2,23177500,1abe268,23177500,232269ec,...) at sleepq_catch_signals+0x368/frame 0x232269b8 sleepq_timedwait_sig(9df42a4,0) at sleepq_timedwait_sig+0x17/frame 0x232269e8 _cv_timedwait_sig_sbt(9df42a4,9df4290,dd23a0ca,4b4,50000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x23226a24 seltdwait(4b4,50000000,0) at seltdwait+0x5e/frame 0x23226a54 kern_select(23177500,5,ffbfd170,0,0,23226bf8,20) at kern_select+0x8ed/frame 0x23226bd4 sys_select(23177500,2317778c) at sys_select+0x41/frame 0x23226c08 syscall(23226ce8,3b,3b,3b,20c0c000,...) at syscall+0x306/frame 0x23226cdc Xint0x80_syscall() at 0xffc033c9/frame 0x23226cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command sshd pid 664 tid 100173 td 0x22fb3e00 sched_switch(22fb3e00,104) at sched_switch+0x4f3/frame 0x207b9970 mi_switch(104) at mi_switch+0x1c4/frame 0x207b99ac sleepq_switch(3440,0,22fb3e00,1da3314,181836a8,...) at sleepq_switch+0xf7/frame 0x207b99cc sleepq_catch_signals(207b9a38,f76998,1837b564,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x207b99f8 sleepq_wait_sig(1837b564,0) at sleepq_wait_sig+0xe/frame 0x207b9a00 _cv_wait_sig(1837b564,1837b550) at _cv_wait_sig+0x138/frame 0x207b9a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x207b9a54 kern_select(22fb3e00,7,2047a180,0,0,0,20) at kern_select+0x8ed/frame 0x207b9bd4 sys_select(22fb3e00,22fb408c) at sys_select+0x41/frame 0x207b9c08 syscall(207b9ce8,3b,3b,3b,20480000,...) at syscall+0x306/frame 0x207b9cdc Xint0x80_syscall() at 0xffc033c9/frame 0x207b9cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command smartd pid 624 tid 100129 td 0x18871880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,38,a07e50,38,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(38,a07e50,38,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(18871880,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command powerd pid 600 tid 100126 td 0x1fc04380 sched_switch(1fc04380,104) at sched_switch+0x4f3/frame 0x20622930 mi_switch(104) at mi_switch+0x1c4/frame 0x2062296c sleepq_switch(29c0,0,1dced30,1da2894,18dfe6a8,...) at sleepq_switch+0xf7/frame 0x2062298c sleepq_catch_signals(e2,1fc04380,1abe268,1fc04380,206229ec,...) at sleepq_catch_signals+0x368/frame 0x206229b8 sleepq_timedwait_sig(9df43e4,0) at sleepq_timedwait_sig+0x17/frame 0x206229e8 _cv_timedwait_sig_sbt(9df43e4,9df43d0,afd438b4,4b0,4000000,0,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20622a24 seltdwait(4b0,4000000,0) at seltdwait+0x5e/frame 0x20622a54 kern_select(1fc04380,0,ffbfed88,0,ffbfed88,20622bf8,20) at kern_select+0x8ed/frame 0x20622bd4 sys_select(1fc04380,1fc0460c) at sys_select+0x41/frame 0x20622c08 syscall(20622ce8,3b,3b,3b,ffbfee08,...) at syscall+0x306/frame 0x20622cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20622cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command ntpd pid 597 tid 100170 td 0x22fb4180 sched_switch(22fb4180,104) at sched_switch+0x4f3/frame 0x20756970 mi_switch(104) at mi_switch+0x1c4/frame 0x207569ac sleepq_switch(440,0,22fb4180,1da0314,181839fc,...) at sleepq_switch+0xf7/frame 0x207569cc sleepq_catch_signals(20756a38,f76998,18cef5e4,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x207569f8 sleepq_wait_sig(18cef5e4,0) at sleepq_wait_sig+0xe/frame 0x20756a00 _cv_wait_sig(18cef5e4,18cef5d0) at _cv_wait_sig+0x138/frame 0x20756a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20756a54 kern_select(22fb4180,1b,ffbfe2c0,0,0,0,20) at kern_select+0x8ed/frame 0x20756bd4 sys_select(22fb4180,22fb440c) at sys_select+0x41/frame 0x20756c08 syscall(20756ce8,3b,3b,3b,20994000,...) at syscall+0x306/frame 0x20756cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20756cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command rpc.lockd pid 559 tid 100136 td 0x22e05380 sched_switch(22e05380,104) at sched_switch+0x4f3/frame 0x206a88e0 mi_switch(104) at mi_switch+0x1c4/frame 0x206a891c sleepq_switch(2200,0,1dcede0,1da20d4,181840a4,...) at sleepq_switch+0xf7/frame 0x206a893c sleepq_catch_signals(e2,22e05380,1abe268,22e05380,206a899c,...) at sleepq_catch_signals+0x368/frame 0x206a8968 sleepq_timedwait_sig(205458d0,0) at sleepq_timedwait_sig+0x17/frame 0x206a8998 _cv_timedwait_sig_sbt(205458d0,1fdf60c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x206a89d4 svc_run_internal(4,1fdf6000,206a8c08,123a215,9c54980,...) at svc_run_internal+0x97d/frame 0x206a8ab4 svc_run(1fdf6000) at svc_run+0x192/frame 0x206a8ae4 sys_nlm_syscall(22e05380,22e0560c) at sys_nlm_syscall+0x625/frame 0x206a8c08 syscall(206a8ce8,3b,3b,3b,20738000,...) at syscall+0x306/frame 0x206a8cdc Xint0x80_syscall() at 0xffc033c9/frame 0x206a8cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command rpc.statd pid 556 tid 100169 td 0x22fb4500 sched_switch(22fb4500,104) at sched_switch+0x4f3/frame 0x2074d930 mi_switch(104) at mi_switch+0x1c4/frame 0x2074d96c sleepq_switch(480,0,1dcee38,1da0354,18183d50,...) at sleepq_switch+0xf7/frame 0x2074d98c sleepq_catch_signals(e2,22fb4500,1abe268,22fb4500,2074d9ec,...) at sleepq_catch_signals+0x368/frame 0x2074d9b8 sleepq_timedwait_sig(1837b6a4,0) at sleepq_timedwait_sig+0x17/frame 0x2074d9e8 _cv_timedwait_sig_sbt(1837b6a4,1837b690,65d82137,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x2074da24 seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x2074da54 kern_select(22fb4500,8,ffbfecf8,0,0,2074dbf8,20) at kern_select+0x8ed/frame 0x2074dbd4 sys_select(22fb4500,22fb478c) at sys_select+0x41/frame 0x2074dc08 syscall(2074dce8,3b,3b,3b,ffbfecf8,...) at syscall+0x306/frame 0x2074dcdc Xint0x80_syscall() at 0xffc033c9/frame 0x2074dcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsd pid 555 tid 100137 td 0x18871500 sched_switch(18871500,104) at sched_switch+0x4f3/frame 0x206ad440 mi_switch(104) at mi_switch+0x1c4/frame 0x206ad47c sleepq_switch(240,0,1dcf938,1da0114,18da374c,...) at sleepq_switch+0xf7/frame 0x206ad49c sleepq_catch_signals(e2,18871500,1abe268,18871500,206ad4fc,...) at sleepq_catch_signals+0x368/frame 0x206ad4c8 sleepq_timedwait_sig(20545950,0) at sleepq_timedwait_sig+0x17/frame 0x206ad4f8 _cv_timedwait_sig_sbt(20545950,186510c0,fffffa38,4,0,0,100) at _cv_timedwait_sig_sbt+0x15a/frame 0x206ad534 svc_run_internal(22fb4880,186510c0,186510d0,1865111c,0,...) at svc_run_internal+0x97d/frame 0x206ad618 svc_run(18651000) at svc_run+0x192/frame 0x206ad648 nfsrvd_nfsd(18871500,206ad7d0) at nfsrvd_nfsd+0x2ec/frame 0x206ad778 nfssvc_nfsd(18871500,1887178c) at nfssvc_nfsd+0x4ea/frame 0x206adbf0 sys_nfssvc(18871500,1887178c) at sys_nfssvc+0x92/frame 0x206adc08 syscall(206adce8,3b,3b,3b,ffbfeba0,...) at syscall+0x306/frame 0x206adcdc Xint0x80_syscall() at 0xffc033c9/frame 0x206adcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsd pid 555 tid 100138 td 0x22e05000 sched_switch(22e05000,104) at sched_switch+0x4f3/frame 0x206b2b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206b2b54 sleepq_switch(25c0,0,22e05000,1da2494,18da374c,...) at sleepq_switch+0xf7/frame 0x206b2b74 sleepq_catch_signals(206b2be0,f76998,1888c750,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206b2ba0 sleepq_wait_sig(1888c750,0) at sleepq_wait_sig+0xe/frame 0x206b2ba8 _cv_wait_sig(1888c750,186510c0) at _cv_wait_sig+0x138/frame 0x206b2be0 svc_run_internal(206b2cd4,f98ac9,186510c0,206b2ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206b2cac svc_thread_start(186510c0,206b2ce8) at svc_thread_start+0xd/frame 0x206b2cb4 fork_exit(12559b0,186510c0,206b2ce8,3b,3b,...) at fork_exit+0x69/frame 0x206b2cd4 fork_trampoline() at 0xffc033de/frame 0x206b2cd4 --- kthread start Tracing command nfsd pid 555 tid 100139 td 0x22e04c00 sched_switch(22e04c00,104) at sched_switch+0x4f3/frame 0x206b7b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206b7b54 sleepq_switch(580,0,22e04c00,1da0454,18da374c,...) at sleepq_switch+0xf7/frame 0x206b7b74 sleepq_catch_signals(206b7be0,f76998,1888c6d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206b7ba0 sleepq_wait_sig(1888c6d0,0) at sleepq_wait_sig+0xe/frame 0x206b7ba8 _cv_wait_sig(1888c6d0,186510c0) at _cv_wait_sig+0x138/frame 0x206b7be0 svc_run_internal(206b7cd4,f98ac9,186510c0,206b7ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206b7cac svc_thread_start(186510c0,206b7ce8) at svc_thread_start+0xd/frame 0x206b7cb4 fork_exit(12559b0,186510c0,206b7ce8,3b,3b,...) at fork_exit+0x69/frame 0x206b7cd4 fork_trampoline() at 0xffc033de/frame 0x206b7cd4 --- kthread start Tracing command nfsd pid 555 tid 100140 td 0x22e04880 sched_switch(22e04880,104) at sched_switch+0x4f3/frame 0x206bcb18 mi_switch(104) at mi_switch+0x1c4/frame 0x206bcb54 sleepq_switch(2b80,0,22e04880,1da2a54,18da374c,...) at sleepq_switch+0xf7/frame 0x206bcb74 sleepq_catch_signals(206bcbe0,f76998,18d0fe50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206bcba0 sleepq_wait_sig(18d0fe50,0) at sleepq_wait_sig+0xe/frame 0x206bcba8 _cv_wait_sig(18d0fe50,186510c0) at _cv_wait_sig+0x138/frame 0x206bcbe0 svc_run_internal(206bccd4,f98ac9,186510c0,206bcce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206bccac svc_thread_start(186510c0,206bcce8) at svc_thread_start+0xd/frame 0x206bccb4 fork_exit(12559b0,186510c0,206bcce8,3b,3b,...) at fork_exit+0x69/frame 0x206bccd4 fork_trampoline() at 0xffc033de/frame 0x206bccd4 --- kthread start Tracing command nfsd pid 555 tid 100141 td 0x22e04500 sched_switch(22e04500,104) at sched_switch+0x4f3/frame 0x206c1b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206c1b54 sleepq_switch(2580,0,22e04500,1da2454,18da374c,...) at sleepq_switch+0xf7/frame 0x206c1b74 sleepq_catch_signals(206c1be0,f76998,1888c650,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206c1ba0 sleepq_wait_sig(1888c650,0) at sleepq_wait_sig+0xe/frame 0x206c1ba8 _cv_wait_sig(1888c650,186510c0) at _cv_wait_sig+0x138/frame 0x206c1be0 svc_run_internal(206c1cd4,f98ac9,186510c0,206c1ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206c1cac svc_thread_start(186510c0,206c1ce8) at svc_thread_start+0xd/frame 0x206c1cb4 fork_exit(12559b0,186510c0,206c1ce8,3b,3b,...) at fork_exit+0x69/frame 0x206c1cd4 fork_trampoline() at 0xffc033de/frame 0x206c1cd4 --- kthread start Tracing command nfsd pid 555 tid 100142 td 0x22e04180 sched_switch(22e04180,104) at sched_switch+0x4f3/frame 0x206c6b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206c6b54 sleepq_switch(540,0,22e04180,1da0414,18da374c,...) at sleepq_switch+0xf7/frame 0x206c6b74 sleepq_catch_signals(206c6be0,f76998,1888c5d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206c6ba0 sleepq_wait_sig(1888c5d0,0) at sleepq_wait_sig+0xe/frame 0x206c6ba8 _cv_wait_sig(1888c5d0,186510c0) at _cv_wait_sig+0x138/frame 0x206c6be0 svc_run_internal(206c6cd4,f98ac9,186510c0,206c6ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206c6cac svc_thread_start(186510c0,206c6ce8) at svc_thread_start+0xd/frame 0x206c6cb4 fork_exit(12559b0,186510c0,206c6ce8,3b,3b,...) at fork_exit+0x69/frame 0x206c6cd4 fork_trampoline() at 0xffc033de/frame 0x206c6cd4 --- kthread start Tracing command nfsd pid 555 tid 100143 td 0x22e03e00 sched_switch(22e03e00,104) at sched_switch+0x4f3/frame 0x206cbb18 mi_switch(104) at mi_switch+0x1c4/frame 0x206cbb54 sleepq_switch(2540,0,22e03e00,1da2414,18da374c,...) at sleepq_switch+0xf7/frame 0x206cbb74 sleepq_catch_signals(206cbbe0,f76998,1888c550,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206cbba0 sleepq_wait_sig(1888c550,0) at sleepq_wait_sig+0xe/frame 0x206cbba8 _cv_wait_sig(1888c550,186510c0) at _cv_wait_sig+0x138/frame 0x206cbbe0 svc_run_internal(206cbcd4,f98ac9,186510c0,206cbce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206cbcac svc_thread_start(186510c0,206cbce8) at svc_thread_start+0xd/frame 0x206cbcb4 fork_exit(12559b0,186510c0,206cbce8,3b,3b,...) at fork_exit+0x69/frame 0x206cbcd4 fork_trampoline() at 0xffc033de/frame 0x206cbcd4 --- kthread start Tracing command nfsd pid 555 tid 100144 td 0x22e03a80 sched_switch(22e03a80,104) at sched_switch+0x4f3/frame 0x206d0b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206d0b54 sleepq_switch(500,0,22e03a80,1da03d4,18da374c,...) at sleepq_switch+0xf7/frame 0x206d0b74 sleepq_catch_signals(206d0be0,f76998,1888c4d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206d0ba0 sleepq_wait_sig(1888c4d0,0) at sleepq_wait_sig+0xe/frame 0x206d0ba8 _cv_wait_sig(1888c4d0,186510c0) at _cv_wait_sig+0x138/frame 0x206d0be0 svc_run_internal(206d0cd4,f98ac9,186510c0,206d0ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206d0cac svc_thread_start(186510c0,206d0ce8) at svc_thread_start+0xd/frame 0x206d0cb4 fork_exit(12559b0,186510c0,206d0ce8,3b,3b,...) at fork_exit+0x69/frame 0x206d0cd4 fork_trampoline() at 0xffc033de/frame 0x206d0cd4 --- kthread start Tracing command nfsd pid 555 tid 100145 td 0x22e03700 sched_switch(22e03700,104) at sched_switch+0x4f3/frame 0x206d5b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206d5b54 sleepq_switch(5c0,0,22e03700,1da0494,18da374c,...) at sleepq_switch+0xf7/frame 0x206d5b74 sleepq_catch_signals(206d5be0,f76998,1888c7d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206d5ba0 sleepq_wait_sig(1888c7d0,0) at sleepq_wait_sig+0xe/frame 0x206d5ba8 _cv_wait_sig(1888c7d0,186510c0) at _cv_wait_sig+0x138/frame 0x206d5be0 svc_run_internal(206d5cd4,f98ac9,186510c0,206d5ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206d5cac svc_thread_start(186510c0,206d5ce8) at svc_thread_start+0xd/frame 0x206d5cb4 fork_exit(12559b0,186510c0,206d5ce8,3b,3b,...) at fork_exit+0x69/frame 0x206d5cd4 fork_trampoline() at 0xffc033de/frame 0x206d5cd4 --- kthread start Tracing command nfsd pid 555 tid 100146 td 0x22e03380 sched_switch(22e03380,104) at sched_switch+0x4f3/frame 0x206dab18 mi_switch(104) at mi_switch+0x1c4/frame 0x206dab54 sleepq_switch(2500,0,22e03380,1da23d4,18da374c,...) at sleepq_switch+0xf7/frame 0x206dab74 sleepq_catch_signals(206dabe0,f76998,1888c450,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206daba0 sleepq_wait_sig(1888c450,0) at sleepq_wait_sig+0xe/frame 0x206daba8 _cv_wait_sig(1888c450,186510c0) at _cv_wait_sig+0x138/frame 0x206dabe0 svc_run_internal(206dacd4,f98ac9,186510c0,206dace8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206dacac svc_thread_start(186510c0,206dace8) at svc_thread_start+0xd/frame 0x206dacb4 fork_exit(12559b0,186510c0,206dace8,3b,3b,...) at fork_exit+0x69/frame 0x206dacd4 fork_trampoline() at 0xffc033de/frame 0x206dacd4 --- kthread start Tracing command nfsd pid 555 tid 100147 td 0x22e03000 sched_switch(22e03000,104) at sched_switch+0x4f3/frame 0x206dfb18 mi_switch(104) at mi_switch+0x1c4/frame 0x206dfb54 sleepq_switch(4c0,0,22e03000,1da0394,18da374c,...) at sleepq_switch+0xf7/frame 0x206dfb74 sleepq_catch_signals(206dfbe0,f76998,1888c3d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206dfba0 sleepq_wait_sig(1888c3d0,0) at sleepq_wait_sig+0xe/frame 0x206dfba8 _cv_wait_sig(1888c3d0,186510c0) at _cv_wait_sig+0x138/frame 0x206dfbe0 svc_run_internal(206dfcd4,f98ac9,186510c0,206dfce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206dfcac svc_thread_start(186510c0,206dfce8) at svc_thread_start+0xd/frame 0x206dfcb4 fork_exit(12559b0,186510c0,206dfce8,3b,3b,...) at fork_exit+0x69/frame 0x206dfcd4 fork_trampoline() at 0xffc033de/frame 0x206dfcd4 --- kthread start Tracing command nfsd pid 555 tid 100148 td 0x201bac00 sched_switch(201bac00,104) at sched_switch+0x4f3/frame 0x206e4b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206e4b54 sleepq_switch(1bc0,0,201bac00,1da1a94,18da374c,...) at sleepq_switch+0xf7/frame 0x206e4b74 sleepq_catch_signals(206e4be0,f76998,9933f50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206e4ba0 sleepq_wait_sig(9933f50,0) at sleepq_wait_sig+0xe/frame 0x206e4ba8 _cv_wait_sig(9933f50,186510c0) at _cv_wait_sig+0x138/frame 0x206e4be0 svc_run_internal(206e4cd4,f98ac9,186510c0,206e4ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206e4cac svc_thread_start(186510c0,206e4ce8) at svc_thread_start+0xd/frame 0x206e4cb4 fork_exit(12559b0,186510c0,206e4ce8,3b,3b,...) at fork_exit+0x69/frame 0x206e4cd4 fork_trampoline() at 0xffc033de/frame 0x206e4cd4 --- kthread start Tracing command nfsd pid 555 tid 100149 td 0x201ba880 sched_switch(201ba880,104) at sched_switch+0x4f3/frame 0x206e9b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206e9b54 sleepq_switch(24c0,0,201ba880,1da2394,18da374c,...) at sleepq_switch+0xf7/frame 0x206e9b74 sleepq_catch_signals(206e9be0,f76998,1888c350,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206e9ba0 sleepq_wait_sig(1888c350,0) at sleepq_wait_sig+0xe/frame 0x206e9ba8 _cv_wait_sig(1888c350,186510c0) at _cv_wait_sig+0x138/frame 0x206e9be0 svc_run_internal(206e9cd4,f98ac9,186510c0,206e9ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206e9cac svc_thread_start(186510c0,206e9ce8) at svc_thread_start+0xd/frame 0x206e9cb4 fork_exit(12559b0,186510c0,206e9ce8,3b,3b,...) at fork_exit+0x69/frame 0x206e9cd4 fork_trampoline() at 0xffc033de/frame 0x206e9cd4 --- kthread start Tracing command nfsd pid 555 tid 100150 td 0x22fb1700 sched_switch(22fb1700,104) at sched_switch+0x4f3/frame 0x206eeb18 mi_switch(104) at mi_switch+0x1c4/frame 0x206eeb54 sleepq_switch(480,0,22fb1700,1da0354,18da374c,...) at sleepq_switch+0xf7/frame 0x206eeb74 sleepq_catch_signals(206eebe0,f76998,1888c2d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206eeba0 sleepq_wait_sig(1888c2d0,0) at sleepq_wait_sig+0xe/frame 0x206eeba8 _cv_wait_sig(1888c2d0,186510c0) at _cv_wait_sig+0x138/frame 0x206eebe0 svc_run_internal(206eecd4,f98ac9,186510c0,206eece8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206eecac svc_thread_start(186510c0,206eece8) at svc_thread_start+0xd/frame 0x206eecb4 fork_exit(12559b0,186510c0,206eece8,3b,3b,...) at fork_exit+0x69/frame 0x206eecd4 fork_trampoline() at 0xffc033de/frame 0x206eecd4 --- kthread start Tracing command nfsd pid 555 tid 100151 td 0x22fb1380 sched_switch(22fb1380,104) at sched_switch+0x4f3/frame 0x206f3b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206f3b54 sleepq_switch(3b80,0,22fb1380,1da3a54,18da374c,...) at sleepq_switch+0xf7/frame 0x206f3b74 sleepq_catch_signals(206f3be0,f76998,9933ed0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206f3ba0 sleepq_wait_sig(9933ed0,0) at sleepq_wait_sig+0xe/frame 0x206f3ba8 _cv_wait_sig(9933ed0,186510c0) at _cv_wait_sig+0x138/frame 0x206f3be0 svc_run_internal(206f3cd4,f98ac9,186510c0,206f3ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206f3cac svc_thread_start(186510c0,206f3ce8) at svc_thread_start+0xd/frame 0x206f3cb4 fork_exit(12559b0,186510c0,206f3ce8,3b,3b,...) at fork_exit+0x69/frame 0x206f3cd4 fork_trampoline() at 0xffc033de/frame 0x206f3cd4 --- kthread start Tracing command nfsd pid 555 tid 100152 td 0x22fb1000 sched_switch(22fb1000,104) at sched_switch+0x4f3/frame 0x206f8b18 mi_switch(104) at mi_switch+0x1c4/frame 0x206f8b54 sleepq_switch(2480,0,22fb1000,1da2354,18da374c,...) at sleepq_switch+0xf7/frame 0x206f8b74 sleepq_catch_signals(206f8be0,f76998,1888c250,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206f8ba0 sleepq_wait_sig(1888c250,0) at sleepq_wait_sig+0xe/frame 0x206f8ba8 _cv_wait_sig(1888c250,186510c0) at _cv_wait_sig+0x138/frame 0x206f8be0 svc_run_internal(206f8cd4,f98ac9,186510c0,206f8ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206f8cac svc_thread_start(186510c0,206f8ce8) at svc_thread_start+0xd/frame 0x206f8cb4 fork_exit(12559b0,186510c0,206f8ce8,3b,3b,...) at fork_exit+0x69/frame 0x206f8cd4 fork_trampoline() at 0xffc033de/frame 0x206f8cd4 --- kthread start Tracing command nfsd pid 555 tid 100153 td 0x22fb0c00 sched_switch(22fb0c00,104) at sched_switch+0x4f3/frame 0x206fdb18 mi_switch(104) at mi_switch+0x1c4/frame 0x206fdb54 sleepq_switch(1b80,0,22fb0c00,1da1a54,18da374c,...) at sleepq_switch+0xf7/frame 0x206fdb74 sleepq_catch_signals(206fdbe0,f76998,9933e50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x206fdba0 sleepq_wait_sig(9933e50,0) at sleepq_wait_sig+0xe/frame 0x206fdba8 _cv_wait_sig(9933e50,186510c0) at _cv_wait_sig+0x138/frame 0x206fdbe0 svc_run_internal(206fdcd4,f98ac9,186510c0,206fdce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x206fdcac svc_thread_start(186510c0,206fdce8) at svc_thread_start+0xd/frame 0x206fdcb4 fork_exit(12559b0,186510c0,206fdce8,3b,3b,...) at fork_exit+0x69/frame 0x206fdcd4 fork_trampoline() at 0xffc033de/frame 0x206fdcd4 --- kthread start Tracing command nfsd pid 555 tid 100154 td 0x22fb0880 sched_switch(22fb0880,104) at sched_switch+0x4f3/frame 0x20702b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20702b54 sleepq_switch(440,0,22fb0880,1da0314,18da374c,...) at sleepq_switch+0xf7/frame 0x20702b74 sleepq_catch_signals(20702be0,f76998,1888c1d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20702ba0 sleepq_wait_sig(1888c1d0,0) at sleepq_wait_sig+0xe/frame 0x20702ba8 _cv_wait_sig(1888c1d0,186510c0) at _cv_wait_sig+0x138/frame 0x20702be0 svc_run_internal(20702cd4,f98ac9,186510c0,20702ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20702cac svc_thread_start(186510c0,20702ce8) at svc_thread_start+0xd/frame 0x20702cb4 fork_exit(12559b0,186510c0,20702ce8,3b,3b,...) at fork_exit+0x69/frame 0x20702cd4 fork_trampoline() at 0xffc033de/frame 0x20702cd4 --- kthread start Tracing command nfsd pid 555 tid 100155 td 0x22fb0500 sched_switch(22fb0500,104) at sched_switch+0x4f3/frame 0x20707b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20707b54 sleepq_switch(3b40,0,22fb0500,1da3a14,18da374c,...) at sleepq_switch+0xf7/frame 0x20707b74 sleepq_catch_signals(20707be0,f76998,9933dd0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20707ba0 sleepq_wait_sig(9933dd0,0) at sleepq_wait_sig+0xe/frame 0x20707ba8 _cv_wait_sig(9933dd0,186510c0) at _cv_wait_sig+0x138/frame 0x20707be0 svc_run_internal(20707cd4,f98ac9,186510c0,20707ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20707cac svc_thread_start(186510c0,20707ce8) at svc_thread_start+0xd/frame 0x20707cb4 fork_exit(12559b0,186510c0,20707ce8,3b,3b,...) at fork_exit+0x69/frame 0x20707cd4 fork_trampoline() at 0xffc033de/frame 0x20707cd4 --- kthread start Tracing command nfsd pid 555 tid 100156 td 0x22fb0180 sched_switch(22fb0180,104) at sched_switch+0x4f3/frame 0x2070cb18 mi_switch(104) at mi_switch+0x1c4/frame 0x2070cb54 sleepq_switch(2440,0,22fb0180,1da2314,18da374c,...) at sleepq_switch+0xf7/frame 0x2070cb74 sleepq_catch_signals(2070cbe0,f76998,1888c150,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2070cba0 sleepq_wait_sig(1888c150,0) at sleepq_wait_sig+0xe/frame 0x2070cba8 _cv_wait_sig(1888c150,186510c0) at _cv_wait_sig+0x138/frame 0x2070cbe0 svc_run_internal(2070ccd4,f98ac9,186510c0,2070cce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2070ccac svc_thread_start(186510c0,2070cce8) at svc_thread_start+0xd/frame 0x2070ccb4 fork_exit(12559b0,186510c0,2070cce8,3b,3b,...) at fork_exit+0x69/frame 0x2070ccd4 fork_trampoline() at 0xffc033de/frame 0x2070ccd4 --- kthread start Tracing command nfsd pid 555 tid 100157 td 0x22fafe00 sched_switch(22fafe00,104) at sched_switch+0x4f3/frame 0x20711b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20711b54 sleepq_switch(1b40,0,22fafe00,1da1a14,18da374c,...) at sleepq_switch+0xf7/frame 0x20711b74 sleepq_catch_signals(20711be0,f76998,9933d50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20711ba0 sleepq_wait_sig(9933d50,0) at sleepq_wait_sig+0xe/frame 0x20711ba8 _cv_wait_sig(9933d50,186510c0) at _cv_wait_sig+0x138/frame 0x20711be0 svc_run_internal(20711cd4,f98ac9,186510c0,20711ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20711cac svc_thread_start(186510c0,20711ce8) at svc_thread_start+0xd/frame 0x20711cb4 fork_exit(12559b0,186510c0,20711ce8,3b,3b,...) at fork_exit+0x69/frame 0x20711cd4 fork_trampoline() at 0xffc033de/frame 0x20711cd4 --- kthread start Tracing command nfsd pid 555 tid 100158 td 0x22fafa80 sched_switch(22fafa80,104) at sched_switch+0x4f3/frame 0x20716b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20716b54 sleepq_switch(400,0,22fafa80,1da02d4,18da374c,...) at sleepq_switch+0xf7/frame 0x20716b74 sleepq_catch_signals(20716be0,f76998,1888c0d0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20716ba0 sleepq_wait_sig(1888c0d0,0) at sleepq_wait_sig+0xe/frame 0x20716ba8 _cv_wait_sig(1888c0d0,186510c0) at _cv_wait_sig+0x138/frame 0x20716be0 svc_run_internal(20716cd4,f98ac9,186510c0,20716ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20716cac svc_thread_start(186510c0,20716ce8) at svc_thread_start+0xd/frame 0x20716cb4 fork_exit(12559b0,186510c0,20716ce8,3b,3b,...) at fork_exit+0x69/frame 0x20716cd4 fork_trampoline() at 0xffc033de/frame 0x20716cd4 --- kthread start Tracing command nfsd pid 555 tid 100159 td 0x22faf700 sched_switch(22faf700,104) at sched_switch+0x4f3/frame 0x2071bb18 mi_switch(104) at mi_switch+0x1c4/frame 0x2071bb54 sleepq_switch(2400,0,22faf700,1da22d4,18da374c,...) at sleepq_switch+0xf7/frame 0x2071bb74 sleepq_catch_signals(2071bbe0,f76998,1888c050,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2071bba0 sleepq_wait_sig(1888c050,0) at sleepq_wait_sig+0xe/frame 0x2071bba8 _cv_wait_sig(1888c050,186510c0) at _cv_wait_sig+0x138/frame 0x2071bbe0 svc_run_internal(2071bcd4,f98ac9,186510c0,2071bce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2071bcac svc_thread_start(186510c0,2071bce8) at svc_thread_start+0xd/frame 0x2071bcb4 fork_exit(12559b0,186510c0,2071bce8,3b,3b,...) at fork_exit+0x69/frame 0x2071bcd4 fork_trampoline() at 0xffc033de/frame 0x2071bcd4 --- kthread start Tracing command nfsd pid 555 tid 100160 td 0x22faf380 sched_switch(22faf380,104) at sched_switch+0x4f3/frame 0x20720b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20720b54 sleepq_switch(3b00,0,22faf380,1da39d4,18da374c,...) at sleepq_switch+0xf7/frame 0x20720b74 sleepq_catch_signals(20720be0,f76998,9933cd0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20720ba0 sleepq_wait_sig(9933cd0,0) at sleepq_wait_sig+0xe/frame 0x20720ba8 _cv_wait_sig(9933cd0,186510c0) at _cv_wait_sig+0x138/frame 0x20720be0 svc_run_internal(20720cd4,f98ac9,186510c0,20720ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20720cac svc_thread_start(186510c0,20720ce8) at svc_thread_start+0xd/frame 0x20720cb4 fork_exit(12559b0,186510c0,20720ce8,3b,3b,...) at fork_exit+0x69/frame 0x20720cd4 fork_trampoline() at 0xffc033de/frame 0x20720cd4 --- kthread start Tracing command nfsd pid 555 tid 100161 td 0x22faf000 sched_switch(22faf000,104) at sched_switch+0x4f3/frame 0x20725b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20725b54 sleepq_switch(3bc0,0,22faf000,1da3a94,18da374c,...) at sleepq_switch+0xf7/frame 0x20725b74 sleepq_catch_signals(20725be0,f76998,1888bf50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20725ba0 sleepq_wait_sig(1888bf50,0) at sleepq_wait_sig+0xe/frame 0x20725ba8 _cv_wait_sig(1888bf50,186510c0) at _cv_wait_sig+0x138/frame 0x20725be0 svc_run_internal(20725cd4,f98ac9,186510c0,20725ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20725cac svc_thread_start(186510c0,20725ce8) at svc_thread_start+0xd/frame 0x20725cb4 fork_exit(12559b0,186510c0,20725ce8,3b,3b,...) at fork_exit+0x69/frame 0x20725cd4 fork_trampoline() at 0xffc033de/frame 0x20725cd4 --- kthread start Tracing command nfsd pid 555 tid 100162 td 0x22e06c00 sched_switch(22e06c00,104) at sched_switch+0x4f3/frame 0x2072ab18 mi_switch(104) at mi_switch+0x1c4/frame 0x2072ab54 sleepq_switch(1b00,0,22e06c00,1da19d4,18da374c,...) at sleepq_switch+0xf7/frame 0x2072ab74 sleepq_catch_signals(2072abe0,f76998,9933c50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2072aba0 sleepq_wait_sig(9933c50,0) at sleepq_wait_sig+0xe/frame 0x2072aba8 _cv_wait_sig(9933c50,186510c0) at _cv_wait_sig+0x138/frame 0x2072abe0 svc_run_internal(2072acd4,f98ac9,186510c0,2072ace8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2072acac svc_thread_start(186510c0,2072ace8) at svc_thread_start+0xd/frame 0x2072acb4 fork_exit(12559b0,186510c0,2072ace8,3b,3b,...) at fork_exit+0x69/frame 0x2072acd4 fork_trampoline() at 0xffc033de/frame 0x2072acd4 --- kthread start Tracing command nfsd pid 555 tid 100163 td 0x22e06880 sched_switch(22e06880,104) at sched_switch+0x4f3/frame 0x2072fb18 mi_switch(104) at mi_switch+0x1c4/frame 0x2072fb54 sleepq_switch(1b80,0,22e06880,1da1a54,18da374c,...) at sleepq_switch+0xf7/frame 0x2072fb74 sleepq_catch_signals(2072fbe0,f76998,1888bed0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2072fba0 sleepq_wait_sig(1888bed0,0) at sleepq_wait_sig+0xe/frame 0x2072fba8 _cv_wait_sig(1888bed0,186510c0) at _cv_wait_sig+0x138/frame 0x2072fbe0 svc_run_internal(2072fcd4,f98ac9,186510c0,2072fce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2072fcac svc_thread_start(186510c0,2072fce8) at svc_thread_start+0xd/frame 0x2072fcb4 fork_exit(12559b0,186510c0,2072fce8,3b,3b,...) at fork_exit+0x69/frame 0x2072fcd4 fork_trampoline() at 0xffc033de/frame 0x2072fcd4 --- kthread start Tracing command nfsd pid 555 tid 100164 td 0x22e06500 sched_switch(22e06500,104) at sched_switch+0x4f3/frame 0x20734b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20734b54 sleepq_switch(3b80,0,22e06500,1da3a54,18da374c,...) at sleepq_switch+0xf7/frame 0x20734b74 sleepq_catch_signals(20734be0,f76998,1888be50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20734ba0 sleepq_wait_sig(1888be50,0) at sleepq_wait_sig+0xe/frame 0x20734ba8 _cv_wait_sig(1888be50,186510c0) at _cv_wait_sig+0x138/frame 0x20734be0 svc_run_internal(20734cd4,f98ac9,186510c0,20734ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20734cac svc_thread_start(186510c0,20734ce8) at svc_thread_start+0xd/frame 0x20734cb4 fork_exit(12559b0,186510c0,20734ce8,3b,3b,...) at fork_exit+0x69/frame 0x20734cd4 fork_trampoline() at 0xffc033de/frame 0x20734cd4 --- kthread start Tracing command nfsd pid 555 tid 100165 td 0x22e06180 sched_switch(22e06180,104) at sched_switch+0x4f3/frame 0x20739b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20739b54 sleepq_switch(3ac0,0,22e06180,1da3994,18da374c,...) at sleepq_switch+0xf7/frame 0x20739b74 sleepq_catch_signals(20739be0,f76998,9933bd0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20739ba0 sleepq_wait_sig(9933bd0,0) at sleepq_wait_sig+0xe/frame 0x20739ba8 _cv_wait_sig(9933bd0,186510c0) at _cv_wait_sig+0x138/frame 0x20739be0 svc_run_internal(20739cd4,f98ac9,186510c0,20739ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20739cac svc_thread_start(186510c0,20739ce8) at svc_thread_start+0xd/frame 0x20739cb4 fork_exit(12559b0,186510c0,20739ce8,3b,3b,...) at fork_exit+0x69/frame 0x20739cd4 fork_trampoline() at 0xffc033de/frame 0x20739cd4 --- kthread start Tracing command nfsd pid 555 tid 100166 td 0x22fb5000 sched_switch(22fb5000,104) at sched_switch+0x4f3/frame 0x2073eb18 mi_switch(104) at mi_switch+0x1c4/frame 0x2073eb54 sleepq_switch(1b40,0,22fb5000,1da1a14,18da374c,...) at sleepq_switch+0xf7/frame 0x2073eb74 sleepq_catch_signals(2073ebe0,f76998,1888bdd0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x2073eba0 sleepq_wait_sig(1888bdd0,0) at sleepq_wait_sig+0xe/frame 0x2073eba8 _cv_wait_sig(1888bdd0,186510c0) at _cv_wait_sig+0x138/frame 0x2073ebe0 svc_run_internal(2073ecd4,f98ac9,186510c0,2073ece8,f98a60,...) at svc_run_internal+0xa7c/frame 0x2073ecac svc_thread_start(186510c0,2073ece8) at svc_thread_start+0xd/frame 0x2073ecb4 fork_exit(12559b0,186510c0,2073ece8,3b,3b,...) at fork_exit+0x69/frame 0x2073ecd4 fork_trampoline() at 0xffc033de/frame 0x2073ecd4 --- kthread start Tracing command nfsd pid 555 tid 100167 td 0x22fb4c00 sched_switch(22fb4c00,104) at sched_switch+0x4f3/frame 0x20743b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20743b54 sleepq_switch(1ac0,0,22fb4c00,1da1994,18da374c,...) at sleepq_switch+0xf7/frame 0x20743b74 sleepq_catch_signals(20743be0,f76998,9933b50,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20743ba0 sleepq_wait_sig(9933b50,0) at sleepq_wait_sig+0xe/frame 0x20743ba8 _cv_wait_sig(9933b50,186510c0) at _cv_wait_sig+0x138/frame 0x20743be0 svc_run_internal(20743cd4,f98ac9,186510c0,20743ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20743cac svc_thread_start(186510c0,20743ce8) at svc_thread_start+0xd/frame 0x20743cb4 fork_exit(12559b0,186510c0,20743ce8,3b,3b,...) at fork_exit+0x69/frame 0x20743cd4 fork_trampoline() at 0xffc033de/frame 0x20743cd4 --- kthread start Tracing command nfsd pid 555 tid 100168 td 0x22fb4880 sched_switch(22fb4880,104) at sched_switch+0x4f3/frame 0x20748b18 mi_switch(104) at mi_switch+0x1c4/frame 0x20748b54 sleepq_switch(3a80,0,22fb4880,1da3954,18da374c,...) at sleepq_switch+0xf7/frame 0x20748b74 sleepq_catch_signals(20748be0,f76998,9933ad0,0,14ad807,...) at sleepq_catch_signals+0x368/frame 0x20748ba0 sleepq_wait_sig(9933ad0,0) at sleepq_wait_sig+0xe/frame 0x20748ba8 _cv_wait_sig(9933ad0,186510c0) at _cv_wait_sig+0x138/frame 0x20748be0 svc_run_internal(20748cd4,f98ac9,186510c0,20748ce8,f98a60,...) at svc_run_internal+0xa7c/frame 0x20748cac svc_thread_start(186510c0,20748ce8) at svc_thread_start+0xd/frame 0x20748cb4 fork_exit(12559b0,186510c0,20748ce8,3b,3b,...) at fork_exit+0x69/frame 0x20748cd4 fork_trampoline() at 0xffc033de/frame 0x20748cd4 --- kthread start Tracing command nfsd pid 552 tid 100135 td 0x22e05700 sched_switch(22e05700,104) at sched_switch+0x4f3/frame 0x2069f970 mi_switch(104) at mi_switch+0x1c4/frame 0x2069f9ac sleepq_switch(3480,0,22e05700,1da3354,181843f8,...) at sleepq_switch+0xf7/frame 0x2069f9cc sleepq_catch_signals(2069fa38,f76998,18cef624,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x2069f9f8 sleepq_wait_sig(18cef624,0) at sleepq_wait_sig+0xe/frame 0x2069fa00 _cv_wait_sig(18cef624,18cef610) at _cv_wait_sig+0x138/frame 0x2069fa38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x2069fa54 kern_select(22e05700,7,ffbfec58,0,0,0,20) at kern_select+0x8ed/frame 0x2069fbd4 sys_select(22e05700,22e0598c) at sys_select+0x41/frame 0x2069fc08 syscall(2069fce8,3b,3b,3b,ffbfecd8,...) at syscall+0x306/frame 0x2069fcdc Xint0x80_syscall() at 0xffc033c9/frame 0x2069fcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command mountd pid 546 tid 100130 td 0x1fdb6880 sched_switch(1fdb6880,104) at sched_switch+0x4f3/frame 0x20642970 mi_switch(104) at mi_switch+0x1c4/frame 0x206429ac sleepq_switch(800,0,1fdb6880,1da06d4,18da174c,...) at sleepq_switch+0xf7/frame 0x206429cc sleepq_catch_signals(20642a38,f76998,9df4464,0,1442eb5,...) at sleepq_catch_signals+0x368/frame 0x206429f8 sleepq_wait_sig(9df4464,0) at sleepq_wait_sig+0xe/frame 0x20642a00 _cv_wait_sig(9df4464,9df4450) at _cv_wait_sig+0x138/frame 0x20642a38 seltdwait(ffffffff,0,0) at seltdwait+0x8a/frame 0x20642a54 kern_select(1fdb6880,a,ffbfe8c8,0,0,0,20) at kern_select+0x8ed/frame 0x20642bd4 sys_select(1fdb6880,1fdb6b0c) at sys_select+0x41/frame 0x20642c08 syscall(20642ce8,3b,3b,3b,ffbfe948,...) at syscall+0x306/frame 0x20642cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20642cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 539 tid 100110 td 0x181ca380 sched_switch(181ca380,104) at sched_switch+0x4f3/frame 0x1faeb930 mi_switch(104) at mi_switch+0x1c4/frame 0x1faeb96c sleepq_switch(1800,0,1dcfa40,1da16d4,181896a8,...) at sleepq_switch+0xf7/frame 0x1faeb98c sleepq_catch_signals(e2,181ca380,1abe268,181ca380,1faeb9ec,...) at sleepq_catch_signals+0x368/frame 0x1faeb9b8 sleepq_timedwait_sig(9df4424,0) at sleepq_timedwait_sig+0x17/frame 0x1faeb9e8 _cv_timedwait_sig_sbt(9df4424,9df4410,5d23bc49,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1faeba24 seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x1faeba54 kern_select(181ca380,4,ffbfdbf0,0,0,1faebbf8,20) at kern_select+0x8ed/frame 0x1faebbd4 sys_select(181ca380,181ca60c) at sys_select+0x41/frame 0x1faebc08 syscall(1faebce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x1faebcdc Xint0x80_syscall() at 0xffc033c9/frame 0x1faebcdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 538 tid 100134 td 0x22e05a80 sched_switch(22e05a80,104) at sched_switch+0x4f3/frame 0x2066e930 mi_switch(104) at mi_switch+0x1c4/frame 0x2066e96c sleepq_switch(1cc0,0,1dcfba0,1da1b94,1818474c,...) at sleepq_switch+0xf7/frame 0x2066e98c sleepq_catch_signals(e2,22e05a80,1abe268,22e05a80,2066e9ec,...) at sleepq_catch_signals+0x368/frame 0x2066e9b8 sleepq_timedwait_sig(9df5724,0) at sleepq_timedwait_sig+0x17/frame 0x2066e9e8 _cv_timedwait_sig_sbt(9df5724,9df5710,5d23bc49,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x2066ea24 seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x2066ea54 kern_select(22e05a80,4,ffbfdbf0,0,0,2066ebf8,20) at kern_select+0x8ed/frame 0x2066ebd4 sys_select(22e05a80,22e05d0c) at sys_select+0x41/frame 0x2066ec08 syscall(2066ece8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x2066ecdc Xint0x80_syscall() at 0xffc033c9/frame 0x2066ecdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 537 tid 100120 td 0x1fc04a80 sched_switch(1fc04a80,104) at sched_switch+0x4f3/frame 0x1fb31930 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb3196c sleepq_switch(1640,0,1dd0178,1da1514,18dfed50,...) at sleepq_switch+0xf7/frame 0x1fb3198c sleepq_catch_signals(e2,1fc04a80,1abe268,1fc04a80,1fb319ec,...) at sleepq_catch_signals+0x368/frame 0x1fb319b8 sleepq_timedwait_sig(18cefda4,0) at sleepq_timedwait_sig+0x17/frame 0x1fb319e8 _cv_timedwait_sig_sbt(18cefda4,18cefd90,4a73797d,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fb31a24 seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x1fb31a54 kern_select(1fc04a80,4,ffbfdbf0,0,0,1fb31bf8,20) at kern_select+0x8ed/frame 0x1fb31bd4 sys_select(1fc04a80,1fc04d0c) at sys_select+0x41/frame 0x1fb31c08 syscall(1fb31ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x1fb31cdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fb31cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 536 tid 100132 td 0x181a4c00 sched_switch(181a4c00,104) at sched_switch+0x4f3/frame 0x20658930 mi_switch(104) at mi_switch+0x1c4/frame 0x2065896c sleepq_switch(3f40,0,1dceee8,1da3e14,18189000,...) at sleepq_switch+0xf7/frame 0x2065898c sleepq_catch_signals(e2,181a4c00,1abe268,181a4c00,206589ec,...) at sleepq_catch_signals+0x368/frame 0x206589b8 sleepq_timedwait_sig(9df59a4,0) at sleepq_timedwait_sig+0x17/frame 0x206589e8 _cv_timedwait_sig_sbt(9df59a4,9df5990,4a73797d,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x20658a24 seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x20658a54 kern_select(181a4c00,4,ffbfdbf0,0,0,20658bf8,20) at kern_select+0x8ed/frame 0x20658bd4 sys_select(181a4c00,181a4e8c) at sys_select+0x41/frame 0x20658c08 syscall(20658ce8,3b,3b,3b,ffbfdbf0,...) at syscall+0x306/frame 0x20658cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20658cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command nfsuserd pid 535 tid 100124 td 0x181ca000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1aa7ebc,1aa7ed4,278a43c0,102bc8c,1aa7ed4,...) at db_trace_self_wrapper+0x2a/frame 0x278a4378 kdb_backtrace(278a44a0,1417ce1,3a,a07e50,3a,...) at kdb_backtrace+0x2e/frame 0x278a43d8 kdb_reenter(3a,a07e50,3a,a07e50,237a50a4,...) at kdb_reenter+0x29/frame 0x278a43e0 trap(278a44ac,8,28,28,ffffffff,...) at trap+0xd1/frame 0x278a44a0 calltrap() at 0xffc0316f/frame 0x278a44a0 --- trap 0xc, eip = 0x13f511a, esp = 0x278a44ec, ebp = 0x278a44f8 --- db_trace_thread(181ca000,ffffffff) at db_trace_thread+0x2a/frame 0x278a44f8 _db_stack_trace_all(1017c05,0,ffffffff,278a4584) at _db_stack_trace_all+0x93/frame 0x278a4558 db_command(1b31f32) at db_command+0x26a/frame 0x278a460c db_script_exec(bd452f,0,278a46b4,278a4698) at db_script_exec+0x1a6/frame 0x278a466c db_command(278a4770,a07d16,0,0,27300e00,...) at db_command+0x26a/frame 0x278a4720 db_command_loop(0,0,27300e00,278a4728,278a4770,...) at db_command_loop+0x3c/frame 0x278a4728 db_trap(3,0) at db_trap+0xf6/frame 0x278a4770 kdb_trap(3,0,278a487c) at kdb_trap+0x119/frame 0x278a47a4 trap(278a487c,8,28,28,100,...) at trap+0x38a/frame 0x278a4870 calltrap() at 0xffc0316f/frame 0x278a4870 --- trap 0x3, eip = 0x1017c05, esp = 0x278a48bc, ebp = 0x278a48c0 --- kdb_enter(1523991,1523991) at kdb_enter+0x35/frame 0x278a48c0 vpanic(1463c56,278a490c,278a490c,278a4948,14188a7,...) at vpanic+0x137/frame 0x278a48ec panic(1463c56,14f2661,200246,0,c,...) at panic+0x14/frame 0x278a4900 trap_fatal(237a50a4,0,278a4984,12edac9,1e4ca50,...) at trap_fatal+0x357/frame 0x278a4948 trap_pfault(187d6000,0,0) at trap_pfault+0x72/frame 0x278a4984 trap(278a4a58,8,28,28,230000,...) at trap+0x3a8/frame 0x278a4a4c calltrap() at 0xffc0316f/frame 0x278a4a4c --- trap 0xc, eip = 0x1046736, esp = 0x278a4a98, ebp = 0x278a4ad8 --- sem_remove(0,18d13e00,1000,2f5,1abe268,...) at sem_remove+0x306/frame 0x278a4ad8 kern_semctl(27300e00,230000,0,0,278a4bc0,...) at kern_semctl+0x3d4/frame 0x278a4b90 sys___semctl(27300e00,2730108c) at sys___semctl+0x153/frame 0x278a4c08 syscall(278a4ce8,3b,3b,3b,0,...) at syscall+0x306/frame 0x278a4cdc Xint0x80_syscall() at 0xffc033c9/frame 0x278a4cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command watchdogd pid 492 tid 100107 td 0x18862000 sched_switch(18862000,104) at sched_switch+0x4f3/frame 0x1fab8a68 mi_switch(104) at mi_switch+0x1c4/frame 0x1fab8aa4 sleepq_switch(3740,6c,3,1da3614,1885ad50,...) at sleepq_switch+0xf7/frame 0x1fab8ac4 sleepq_catch_signals(0,152873e,18862000,100,1d921fc,...) at sleepq_catch_signals+0x368/frame 0x1fab8af0 sleepq_timedwait_sig(1d921fc,6c,0,1442eb5,32d,...) at sleepq_timedwait_sig+0x17/frame 0x1fab8b1c _sleep(1d921fc,0,16c,153b48a,2761517b,4d8,c0000000,3,200) at _sleep+0x21c/frame 0x1fab8b5c kern_clock_nanosleep(18862000,0,0,1fab8be8,1fab8be0) at kern_clock_nanosleep+0x1b4/frame 0x1fab8bc4 sys_nanosleep(18862000,1886228c) at sys_nanosleep+0x55/frame 0x1fab8c08 syscall(1fab8ce8,3b,3b,3b,3c,...) at syscall+0x306/frame 0x1fab8cdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fab8cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command rpcbind pid 403 tid 100114 td 0x1fdb6c00 sched_switch(1fdb6c00,104) at sched_switch+0x4f3/frame 0x1fb13958 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb13994 sleepq_switch(2940,0,1dcfca8,1da2814,18da1aa0,...) at sleepq_switch+0xf7/frame 0x1fb139b4 sleepq_catch_signals(e2,1fdb6c00,1abe268,1fdb6c00,1fb13a14,...) at sleepq_catch_signals+0x368/frame 0x1fb139e0 sleepq_timedwait_sig(18cf01a4,0) at sleepq_timedwait_sig+0x17/frame 0x1fb13a10 _cv_timedwait_sig_sbt(18cf01a4,18cf0190,fca0e251,4c1,e0000000,1,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fb13a4c seltdwait(4c1,e0000000,1) at seltdwait+0x5e/frame 0x1fb13a7c kern_poll(1fdb6c00,ffbfcdb8,8,1fb13bf0,0) at kern_poll+0x476/frame 0x1fb13bd4 sys_poll(1fdb6c00,1fdb6e8c) at sys_poll+0x50/frame 0x1fb13c08 syscall(1fb13ce8,3b,3b,3b,ffbfcd38,...) at syscall+0x306/frame 0x1fb13cdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fb13cdc --- syscall (510, FreeBSD ELF32, sys___semctl), eip = 0xffc01430, esp = 0xffc07fe8, ebp = 0xffbfe750 --- Tracing command syslogd pid 394 tid 100128 td 0x18871c00 sched_switch(18871c00,104) at sched_switch+0x4f3/frame 0x20634a60 mi_switch(104) at mi_switch+0x1c4/frame 0x20634a9c sleepq_switch(205ec6d0,5d) at sleepq_switch+0xf7/frame 0x20634abc _sleep(205ec6d0,205ec688,5d,153bb65,0,0,0,0,100) at _sleep+0x2f7/frame 0x20634afc bufobj_wwait(205ec688,0,0) at bufobj_wwait+0x93/frame 0x20634b3c ffs_syncvnode(205ec5c0,1,0) at ffs_syncvnode+0x306/frame 0x20634b80 ffs_fsync(20634bc8) at ffs_fsync+0x18/frame 0x20634ba4 VOP_FSYNC_APV(1af7424,20634bc8,1b0d128,205ec5c0,1,...) at VOP_FSYNC_APV+0x4d/frame 0x20634bb8 kern_fsync(18871c00,e,1) at kern_fsync+0x18c/frame 0x20634bf4 sys_fsync(18871c00,18871e8c) at sys_fsync+0x12/frame 0x20634c08 syscall(20634ce8,3b,3b,3b,20671000,...) at syscall+0x306/frame 0x20634cdc Xint0x80_syscall() at 0xffc033c9/frame 0x20634cdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command devd pid 322 tid 100113 td 0x1fc05c00 sched_switch(1fc05c00,104) at sched_switch+0x4f3/frame 0x1fb0a930 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb0a96c sleepq_switch(2a80,0,1dcfdb0,1da2954,18dff0a4,...) at sleepq_switch+0xf7/frame 0x1fb0a98c sleepq_catch_signals(e2,1fc05c00,1abe268,1fc05c00,1fb0a9ec,...) at sleepq_catch_signals+0x368/frame 0x1fb0a9b8 sleepq_timedwait_sig(17ee0ea4,0) at sleepq_timedwait_sig+0x17/frame 0x1fb0a9e8 _cv_timedwait_sig_sbt(17ee0ea4,17ee0e90,94b134ed,4ca,c0000000,3,200) at _cv_timedwait_sig_sbt+0x15a/frame 0x1fb0aa24 seltdwait(4ca,c0000000,3) at seltdwait+0x5e/frame 0x1fb0aa54 kern_select(1fc05c00,6,ffbfcd98,0,0,1fb0abf8,20) at kern_select+0x8ed/frame 0x1fb0abd4 sys_select(1fc05c00,1fc05e8c) at sys_select+0x41/frame 0x1fb0ac08 syscall(1fb0ace8,3b,3b,3b,ffbfce18,...) at syscall+0x306/frame 0x1fb0acdc Xint0x80_syscall() at 0xffc033c9/frame 0x1fb0acdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command vnlru pid 25 tid 100104 td 0x18862380 sched_switch(18862380,104) at sched_switch+0x4f3/frame 0x1bdf1b68 mi_switch(104) at mi_switch+0x1c4/frame 0x1bdf1ba8 sleepq_switch(18862380,0,1885b0a4,1bdf1c2c,18862380,...) at sleepq_switch+0xf7/frame 0x1bdf1bc8 sleepq_timedwait(1885b0a4,60,1bdf1c20,fb6445,14c8c2c,...) at sleepq_timedwait+0x3f/frame 0x1bdf1bec _sleep(1885b0a4,1ad6740,260,154f73c,fffffed8,0,0,0,100) at _sleep+0x22c/frame 0x1bdf1c2c vnlru_proc(0,1bdf1ce8) at vnlru_proc+0xb02/frame 0x1bdf1cb4 fork_exit(109d4d0,0,1bdf1ce8,0,0,...) at fork_exit+0x69/frame 0x1bdf1cd4 fork_trampoline() at 0xffc033de/frame 0x1bdf1cd4 --- kthread start Tracing command syncer pid 24 tid 100103 td 0x18862700 sched_switch(18862700,104) at sched_switch+0x4f3/frame 0x1d3ffb80 mi_switch(104) at mi_switch+0x1c4/frame 0x1d3ffbc0 sleepq_switch(1abe268,18862700,1d3ffc0c,fb56fe,1dff118,...) at sleepq_switch+0xf7/frame 0x1d3ffbe0 sleepq_timedwait(1dff128,0) at sleepq_timedwait+0x3f/frame 0x1d3ffc08 _cv_timedwait_sbt(1dff128,1dff108,fffffed8,0,0,0,100) at _cv_timedwait_sbt+0x157/frame 0x1d3ffc44 sched_sync(0,1d3ffce8) at sched_sync+0x68c/frame 0x1d3ffcb4 fork_exit(109e440,0,1d3ffce8,0,0,...) at fork_exit+0x69/frame 0x1d3ffcd4 fork_trampoline() at 0xffc033de/frame 0x1d3ffcd4 --- kthread start Tracing command bufdaemon pid 23 tid 100099 td 0x18862a80 sched_switch(18862a80,104) at sched_switch+0x4f3/frame 0xb7d8bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7d8bf0 sleepq_switch(18862a80,0,1dfeaec,b7d8c74,18862a80,...) at sleepq_switch+0xf7/frame 0xb7d8c10 sleepq_timedwait(1dfeaec,54,1885b7f4,1885b804,14432e9,...) at sleepq_timedwait+0x3f/frame 0xb7d8c34 _sleep(1dfeaec,1acad40,54,14728e7,1999997c,0,0,0,100) at _sleep+0x22c/frame 0xb7d8c74 buf_daemon(0,b7d8ce8) at buf_daemon+0xf5/frame 0xb7d8cb4 fork_exit(107bc90,0,b7d8ce8,0,0,...) at fork_exit+0x69/frame 0xb7d8cd4 fork_trampoline() at 0xffc033de/frame 0xb7d8cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100102 td 0x181ca700 sched_switch(181ca700,104) at sched_switch+0x4f3/frame 0x1d3faba0 mi_switch(104) at mi_switch+0x1c4/frame 0x1d3fabe0 sleepq_switch(152873e,181ca700,0,1acc040,1d3fac68,...) at sleepq_switch+0xf7/frame 0x1d3fac00 sleepq_timedwait(1acc040,5c,1885b7f4,1885b804,14432e9,...) at sleepq_timedwait+0x3f/frame 0x1d3fac28 _sleep(1acc040,1acbf80,25c,1476273,fffffed8,0,0,0,100) at _sleep+0x22c/frame 0x1d3fac68 bufspace_daemon(1acae40,1d3face8) at bufspace_daemon+0x2db/frame 0x1d3facb4 fork_exit(107bfa0,1acae40,1d3face8,0,0,...) at fork_exit+0x69/frame 0x1d3facd4 fork_trampoline() at 0xffc033de/frame 0x1d3facd4 --- kthread start Tracing command bufdaemon pid 23 tid 100115 td 0x1fc05880 sched_switch(1fc05880,104) at sched_switch+0x4f3/frame 0x1fb18bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb18bf0 sleepq_switch(152873e,1fc05880,0,1fdae084,1fb18c78,...) at sleepq_switch+0xf7/frame 0x1fb18c10 sleepq_timedwait(1fdae084,54,1fb18c6c,fd0030,14ad9f3,...) at sleepq_timedwait+0x3f/frame 0x1fb18c38 _sleep(1fdae084,1fdae000,54,14a428d,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x1fb18c78 softdep_flush(18bfaac0,1fb18ce8) at softdep_flush+0x15a/frame 0x1fb18cb4 fork_exit(1291150,18bfaac0,1fb18ce8,0,0,...) at fork_exit+0x69/frame 0x1fb18cd4 fork_trampoline() at 0xffc033de/frame 0x1fb18cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100116 td 0x1fc05500 sched_switch(1fc05500,104) at sched_switch+0x4f3/frame 0x1fb1dbb0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb1dbf0 sleepq_switch(152873e,1fc05500,0,1fdae484,1fb1dc78,...) at sleepq_switch+0xf7/frame 0x1fb1dc10 sleepq_timedwait(1fdae484,54,1fb1dc6c,fd0030,14ad9f3,...) at sleepq_timedwait+0x3f/frame 0x1fb1dc38 _sleep(1fdae484,1fdae400,54,14a428d,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x1fb1dc78 softdep_flush(1fdcf000,1fb1dce8) at softdep_flush+0x15a/frame 0x1fb1dcb4 fork_exit(1291150,1fdcf000,1fb1dce8,0,0,...) at fork_exit+0x69/frame 0x1fb1dcd4 fork_trampoline() at 0xffc033de/frame 0x1fb1dcd4 --- kthread start Tracing command bufdaemon pid 23 tid 100117 td 0x1fc05180 sched_switch(1fc05180,104) at sched_switch+0x4f3/frame 0x1fb22bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb22bf0 sleepq_switch(152873e,1fc05180,0,9966684,1fb22c78,...) at sleepq_switch+0xf7/frame 0x1fb22c10 sleepq_timedwait(9966684,54,1fb22c6c,fd0030,14ad9f3,...) at sleepq_timedwait+0x3f/frame 0x1fb22c38 _sleep(9966684,9966600,54,14a428d,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x1fb22c78 softdep_flush(18bfe080,1fb22ce8) at softdep_flush+0x15a/frame 0x1fb22cb4 fork_exit(1291150,18bfe080,1fb22ce8,0,0,...) at fork_exit+0x69/frame 0x1fb22cd4 fork_trampoline() at 0xffc033de/frame 0x1fb22cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100118 td 0x1fc04e00 sched_switch(1fc04e00,104) at sched_switch+0x4f3/frame 0x1fb27bb0 mi_switch(104) at mi_switch+0x1c4/frame 0x1fb27bf0 sleepq_switch(152873e,1fc04e00,0,1fdae284,1fb27c78,...) at sleepq_switch+0xf7/frame 0x1fb27c10 sleepq_timedwait(1fdae284,54,1fb27c6c,fd0030,14ad9f3,...) at sleepq_timedwait+0x3f/frame 0x1fb27c38 _sleep(1fdae284,1fdae200,54,14a428d,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x1fb27c78 softdep_flush(18bfd5c0,1fb27ce8) at softdep_flush+0x15a/frame 0x1fb27cb4 fork_exit(1291150,18bfd5c0,1fb27ce8,0,0,...) at fork_exit+0x69/frame 0x1fb27cd4 fork_trampoline() at 0xffc033de/frame 0x1fb27cd4 --- kthread start Tracing command bufdaemon pid 23 tid 100671 td 0x2730c500 sched_switch(2730c500,104) at sched_switch+0x4f3/frame 0x26f5cbb0 mi_switch(104) at mi_switch+0x1c4/frame 0x26f5cbf0 sleepq_switch(152873e,2730c500,0,18d7f284,26f5cc78,...) at sleepq_switch+0xf7/frame 0x26f5cc10 sleepq_timedwait(18d7f284,54,26f5cc6c,fd0030,14ad9f3,...) at sleepq_timedwait+0x3f/frame 0x26f5cc38 _sleep(18d7f284,18d7f200,54,14a428d,7fffff6c,0,0,0,100) at _sleep+0x22c/frame 0x26f5cc78 softdep_flush(1fdd25c0,26f5cce8) at softdep_flush+0x15a/frame 0x26f5ccb4 fork_exit(1291150,1fdd25c0,26f5cce8,0,0,...) at fork_exit+0x69/frame 0x26f5ccd4 fork_trampoline() at 0xffc033de/frame 0x26f5ccd4 --- kthread start Tracing command vmdaemon pid 22 tid 100098 td 0x18862e00 sched_switch(18862e00,104) at sched_switch+0x4f3/frame 0xb7d3ba0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7d3bdc sleepq_switch(1e13e30,74) at sleepq_switch+0xf7/frame 0xb7d3bfc _sleep(1e13e30,1e13e1c,74,14728e0,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7d3c3c vm_daemon(0,b7d3ce8) at vm_daemon+0x7f/frame 0xb7d3cb4 fork_exit(12f9e00,0,b7d3ce8,0,0,...) at fork_exit+0x69/frame 0xb7d3cd4 fork_trampoline() at 0xffc033de/frame 0xb7d3cd4 --- kthread start Tracing command pagedaemon pid 21 tid 100097 td 0x18863180 sched_switch(18863180,104) at sched_switch+0x4f3/frame 0xb7ce9a0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7ce9e0 sleepq_switch(18863180,0,187ab55,b7cea64,18863180,...) at sleepq_switch+0xf7/frame 0xb7cea00 sleepq_timedwait(187ab55,0,b7cea58,fb5d17,1e4c980,...) at sleepq_timedwait+0x3f/frame 0xb7cea24 _sleep(187ab55,0,0,146b7fa,1999997c,0,0,0,100) at _sleep+0x22c/frame 0xb7cea64 pause_sbt(146b7fa,1999997c,0,0,0,100) at pause_sbt+0x87/frame 0xb7ceaa0 vm_pageout_worker(0,18863180,18860000,18863180,18860000,...) at vm_pageout_worker+0x210/frame 0xb7cec94 vm_pageout(0,b7cece8) at vm_pageout+0x146/frame 0xb7cecb4 fork_exit(12ef0b0,0,b7cece8,0,0,...) at fork_exit+0x69/frame 0xb7cecd4 fork_trampoline() at 0xffc033de/frame 0xb7cecd4 --- kthread start Tracing command pagedaemon pid 21 tid 100100 td 0x181cae00 sched_switch(181cae00,104) at sched_switch+0x4f3/frame 0xb7fd7f8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7fd834 sleepq_switch(1e13880,54) at sleepq_switch+0xf7/frame 0xb7fd854 _sleep(1e13880,1e13888,54,14e6800,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7fd894 swap_pager_putpages(26932e18,b7fda7c,20,4,b7fd93c) at swap_pager_putpages+0x19c/frame 0xb7fd920 vm_pageout_flush(b7fda7c,20,4,0,0,0) at vm_pageout_flush+0xbb/frame 0xb7fd9dc vm_pageout_cluster(0,0,0,5,2,...) at vm_pageout_cluster+0x334/frame 0xb7fdb38 vm_pageout_laundry_worker(0,b7fdce8) at vm_pageout_laundry_worker+0x972/frame 0xb7fdcb4 fork_exit(12f0850,0,b7fdce8,0,0,...) at fork_exit+0x69/frame 0xb7fdcd4 fork_trampoline() at 0xffc033de/frame 0xb7fdcd4 --- kthread start Tracing command pagedaemon pid 21 tid 100101 td 0x181caa80 sched_switch(181caa80,100) at sched_switch+0x4f3/frame 0x97fcc18 mi_switch(100) at mi_switch+0x1c4/frame 0x97fcc54 sched_bind(181caa80,0) at sched_bind+0xbf/frame 0x97fcc68 pcpu_cache_drain_safe(98534c0,98534dc,0,97fccb4,12cac8c,...) at pcpu_cache_drain_safe+0x1c4/frame 0x97fcc88 uma_reclaim(2) at uma_reclaim+0xef/frame 0x97fcc9c uma_reclaim_worker(0,97fcce8) at uma_reclaim_worker+0x2c/frame 0x97fccb4 fork_exit(12cac60,0,97fcce8,0,0,...) at fork_exit+0x69/frame 0x97fccd4 fork_trampoline() at 0xffc033de/frame 0x97fccd4 --- kthread start Tracing command rand_harvestq pid 20 tid 100094 td 0x18863c00 sched_switch(18863c00,104) at sched_switch+0x4f3/frame 0xb7bfb78 mi_switch(104) at mi_switch+0x1c4/frame 0xb7bfbb8 sleepq_switch(18863c00,0,1bf607c,b7bfc3c,18863c00,...) at sleepq_switch+0xf7/frame 0xb7bfbd8 sleepq_timedwait(1bf607c,0,1bfb16c,0,b7bfc38,...) at sleepq_timedwait+0x3f/frame 0xb7bfbfc _sleep(1bf607c,0,0,1476273,19999999,0,0,0,4) at _sleep+0x22c/frame 0xb7bfc3c random_kthread(0,b7bfce8) at random_kthread+0x50/frame 0xb7bfcb4 fork_exit(d46790,0,b7bfce8,0,0,...) at fork_exit+0x69/frame 0xb7bfcd4 fork_trampoline() at 0xffc033de/frame 0xb7bfcd4 --- kthread start Tracing command sctp_iterator pid 19 tid 100090 td 0x18870a80 sched_switch(18870a80,104) at sched_switch+0x4f3/frame 0xb7abbe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7abc28 sleepq_switch(1e5dd78,0) at sleepq_switch+0xf7/frame 0xb7abc48 _sleep(1e5dd78,1e5dd40,0,146aab2,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7abc88 sctp_iterator_thread(0,b7abce8) at sctp_iterator_thread+0x43/frame 0xb7abcb4 fork_exit(1160260,0,b7abce8,0,0,...) at fork_exit+0x69/frame 0xb7abcd4 fork_trampoline() at 0xffc033de/frame 0xb7abcd4 --- kthread start Tracing command soaiod4 pid 18 tid 100089 td 0x991e500 sched_switch(991e500,104) at sched_switch+0x4f3/frame 0xb7a1bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7a1bf0 sleepq_switch(152873e,991e500,0,1dfe728,b7a1c78,...) at sleepq_switch+0xf7/frame 0xb7a1c10 sleepq_timedwait(1dfe728,0,0,147c740,14eec8a,...) at sleepq_timedwait+0x3f/frame 0xb7a1c38 _sleep(1dfe728,1dfe708,0,1476273,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb7a1c78 soaio_kproc_loop(4,b7a1ce8) at soaio_kproc_loop+0x148/frame 0xb7a1cb4 fork_exit(10428f0,4,b7a1ce8,0,0,...) at fork_exit+0x69/frame 0xb7a1cd4 fork_trampoline() at 0xffc033de/frame 0xb7a1cd4 --- kthread start Tracing command soaiod3 pid 17 tid 100088 td 0x991e880 sched_switch(991e880,104) at sched_switch+0x4f3/frame 0xb79cbb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb79cbf0 sleepq_switch(152873e,991e880,0,1dfe728,b79cc78,...) at sleepq_switch+0xf7/frame 0xb79cc10 sleepq_timedwait(1dfe728,0,0,147c740,14eec8a,...) at sleepq_timedwait+0x3f/frame 0xb79cc38 _sleep(1dfe728,1dfe708,0,1476273,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb79cc78 soaio_kproc_loop(3,b79cce8) at soaio_kproc_loop+0x148/frame 0xb79ccb4 fork_exit(10428f0,3,b79cce8,0,0,...) at fork_exit+0x69/frame 0xb79ccd4 fork_trampoline() at 0xffc033de/frame 0xb79ccd4 --- kthread start Tracing command soaiod2 pid 16 tid 100087 td 0x991ec00 sched_switch(991ec00,104) at sched_switch+0x4f3/frame 0xb797bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb797bf0 sleepq_switch(152873e,991ec00,0,1dfe728,b797c78,...) at sleepq_switch+0xf7/frame 0xb797c10 sleepq_timedwait(1dfe728,0,0,147c740,14eec8a,...) at sleepq_timedwait+0x3f/frame 0xb797c38 _sleep(1dfe728,1dfe708,0,1476273,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb797c78 soaio_kproc_loop(2,b797ce8) at soaio_kproc_loop+0x148/frame 0xb797cb4 fork_exit(10428f0,2,b797ce8,0,0,...) at fork_exit+0x69/frame 0xb797cd4 fork_trampoline() at 0xffc033de/frame 0xb797cd4 --- kthread start Tracing command soaiod1 pid 9 tid 100086 td 0x9985000 sched_switch(9985000,104) at sched_switch+0x4f3/frame 0xb792bb0 mi_switch(104) at mi_switch+0x1c4/frame 0xb792bf0 sleepq_switch(152873e,9985000,0,1dfe728,b792c78,...) at sleepq_switch+0xf7/frame 0xb792c10 sleepq_timedwait(1dfe728,0,0,147c740,14eec8a,...) at sleepq_timedwait+0x3f/frame 0xb792c38 _sleep(1dfe728,1dfe708,0,1476273,ffffdd50,1d,0,0,100) at _sleep+0x22c/frame 0xb792c78 soaio_kproc_loop(1,b792ce8) at soaio_kproc_loop+0x148/frame 0xb792cb4 fork_exit(10428f0,1,b792ce8,0,0,...) at fork_exit+0x69/frame 0xb792cd4 fork_trampoline() at 0xffc033de/frame 0xb792cd4 --- kthread start Tracing command fdc0 pid 8 tid 100083 td 0x181cb880 sched_switch(181cb880,104) at sched_switch+0x4f3/frame 0xb783ba0 mi_switch(104) at mi_switch+0x1c4/frame 0xb783bdc sleepq_switch(1813023c,5c) at sleepq_switch+0xf7/frame 0xb783bfc _sleep(1813023c,181302f8,5c,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0xb783c3c fdc_thread(18130200,b783ce8) at fdc_thread+0x351/frame 0xb783cb4 fork_exit(1376550,18130200,b783ce8,0,0,...) at fork_exit+0x69/frame 0xb783cd4 fork_trampoline() at 0xffc033de/frame 0xb783cd4 --- kthread start Tracing command usb pid 15 tid 100048 td 0x17ed6180 sched_switch(17ed6180,104) at sched_switch+0x4f3/frame 0xb6cbc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6cbc3c sleepq_switch(17ec2aac,0) at sleepq_switch+0xf7/frame 0xb6cbc5c _cv_wait(17ec2aac,17ec2cd8) at _cv_wait+0x135/frame 0xb6cbc94 usb_process(17ec2aa4,b6cbce8) at usb_process+0x9d/frame 0xb6cbcb4 fork_exit(e3aa70,17ec2aa4,b6cbce8,0,0,...) at fork_exit+0x69/frame 0xb6cbcd4 fork_trampoline() at 0xffc033de/frame 0xb6cbcd4 --- kthread start Tracing command usb pid 15 tid 100049 td 0x17ed5e00 sched_switch(17ed5e00,104) at sched_switch+0x4f3/frame 0xb6d0c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6d0c3c sleepq_switch(17ec2adc,0) at sleepq_switch+0xf7/frame 0xb6d0c5c _cv_wait(17ec2adc,17ec2cd8) at _cv_wait+0x135/frame 0xb6d0c94 usb_process(17ec2ad4,b6d0ce8) at usb_process+0x9d/frame 0xb6d0cb4 fork_exit(e3aa70,17ec2ad4,b6d0ce8,0,0,...) at fork_exit+0x69/frame 0xb6d0cd4 fork_trampoline() at 0xffc033de/frame 0xb6d0cd4 --- kthread start Tracing command usb pid 15 tid 100050 td 0x17ed5a80 sched_switch(17ed5a80,104) at sched_switch+0x4f3/frame 0xb6d5c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6d5c3c sleepq_switch(17ec2b0c,0) at sleepq_switch+0xf7/frame 0xb6d5c5c _cv_wait(17ec2b0c,17ec2cd8) at _cv_wait+0x135/frame 0xb6d5c94 usb_process(17ec2b04,b6d5ce8) at usb_process+0x9d/frame 0xb6d5cb4 fork_exit(e3aa70,17ec2b04,b6d5ce8,0,0,...) at fork_exit+0x69/frame 0xb6d5cd4 fork_trampoline() at 0xffc033de/frame 0xb6d5cd4 --- kthread start Tracing command usb pid 15 tid 100051 td 0x17ed5700 sched_switch(17ed5700,104) at sched_switch+0x4f3/frame 0xb6dac00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6dac3c sleepq_switch(17ec2b3c,0) at sleepq_switch+0xf7/frame 0xb6dac5c _cv_wait(17ec2b3c,17ec2cd8) at _cv_wait+0x135/frame 0xb6dac94 usb_process(17ec2b34,b6dace8) at usb_process+0x9d/frame 0xb6dacb4 fork_exit(e3aa70,17ec2b34,b6dace8,0,0,...) at fork_exit+0x69/frame 0xb6dacd4 fork_trampoline() at 0xffc033de/frame 0xb6dacd4 --- kthread start Tracing command usb pid 15 tid 100052 td 0x17ed5380 sched_switch(17ed5380,104) at sched_switch+0x4f3/frame 0xb6dfc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6dfc3c sleepq_switch(17ec2b6c,0) at sleepq_switch+0xf7/frame 0xb6dfc5c _cv_wait(17ec2b6c,17ec2cd8) at _cv_wait+0x135/frame 0xb6dfc94 usb_process(17ec2b64,b6dfce8) at usb_process+0x9d/frame 0xb6dfcb4 fork_exit(e3aa70,17ec2b64,b6dfce8,0,0,...) at fork_exit+0x69/frame 0xb6dfcd4 fork_trampoline() at 0xffc033de/frame 0xb6dfcd4 --- kthread start Tracing command usb pid 15 tid 100054 td 0x9988c00 sched_switch(9988c00,104) at sched_switch+0x4f3/frame 0xb6eac00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6eac3c sleepq_switch(18197aac,0) at sleepq_switch+0xf7/frame 0xb6eac5c _cv_wait(18197aac,18197cd8) at _cv_wait+0x135/frame 0xb6eac94 usb_process(18197aa4,b6eace8) at usb_process+0x9d/frame 0xb6eacb4 fork_exit(e3aa70,18197aa4,b6eace8,0,0,...) at fork_exit+0x69/frame 0xb6eacd4 fork_trampoline() at 0xffc033de/frame 0xb6eacd4 --- kthread start Tracing command usb pid 15 tid 100055 td 0x9988880 sched_switch(9988880,104) at sched_switch+0x4f3/frame 0xb6efc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6efc3c sleepq_switch(18197adc,0) at sleepq_switch+0xf7/frame 0xb6efc5c _cv_wait(18197adc,18197cd8) at _cv_wait+0x135/frame 0xb6efc94 usb_process(18197ad4,b6efce8) at usb_process+0x9d/frame 0xb6efcb4 fork_exit(e3aa70,18197ad4,b6efce8,0,0,...) at fork_exit+0x69/frame 0xb6efcd4 fork_trampoline() at 0xffc033de/frame 0xb6efcd4 --- kthread start Tracing command usb pid 15 tid 100056 td 0x9988500 sched_switch(9988500,104) at sched_switch+0x4f3/frame 0xb6f4c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6f4c3c sleepq_switch(18197b0c,0) at sleepq_switch+0xf7/frame 0xb6f4c5c _cv_wait(18197b0c,18197cd8) at _cv_wait+0x135/frame 0xb6f4c94 usb_process(18197b04,b6f4ce8) at usb_process+0x9d/frame 0xb6f4cb4 fork_exit(e3aa70,18197b04,b6f4ce8,0,0,...) at fork_exit+0x69/frame 0xb6f4cd4 fork_trampoline() at 0xffc033de/frame 0xb6f4cd4 --- kthread start Tracing command usb pid 15 tid 100057 td 0x9988180 sched_switch(9988180,104) at sched_switch+0x4f3/frame 0xb6f9c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6f9c3c sleepq_switch(18197b3c,0) at sleepq_switch+0xf7/frame 0xb6f9c5c _cv_wait(18197b3c,18197cd8) at _cv_wait+0x135/frame 0xb6f9c94 usb_process(18197b34,b6f9ce8) at usb_process+0x9d/frame 0xb6f9cb4 fork_exit(e3aa70,18197b34,b6f9ce8,0,0,...) at fork_exit+0x69/frame 0xb6f9cd4 fork_trampoline() at 0xffc033de/frame 0xb6f9cd4 --- kthread start Tracing command usb pid 15 tid 100058 td 0x9987e00 sched_switch(9987e00,104) at sched_switch+0x4f3/frame 0xb6fec00 mi_switch(104) at mi_switch+0x1c4/frame 0xb6fec3c sleepq_switch(18197b6c,0) at sleepq_switch+0xf7/frame 0xb6fec5c _cv_wait(18197b6c,18197cd8) at _cv_wait+0x135/frame 0xb6fec94 usb_process(18197b64,b6fece8) at usb_process+0x9d/frame 0xb6fecb4 fork_exit(e3aa70,18197b64,b6fece8,0,0,...) at fork_exit+0x69/frame 0xb6fecd4 fork_trampoline() at 0xffc033de/frame 0xb6fecd4 --- kthread start Tracing command usb pid 15 tid 100060 td 0x181a4880 sched_switch(181a4880,104) at sched_switch+0x4f3/frame 0xb709c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb709c3c sleepq_switch(1819faac,0) at sleepq_switch+0xf7/frame 0xb709c5c _cv_wait(1819faac,1819fcd8) at _cv_wait+0x135/frame 0xb709c94 usb_process(1819faa4,b709ce8) at usb_process+0x9d/frame 0xb709cb4 fork_exit(e3aa70,1819faa4,b709ce8,0,0,...) at fork_exit+0x69/frame 0xb709cd4 fork_trampoline() at 0xffc033de/frame 0xb709cd4 --- kthread start Tracing command usb pid 15 tid 100061 td 0x181a4500 sched_switch(181a4500,104) at sched_switch+0x4f3/frame 0xb70ec00 mi_switch(104) at mi_switch+0x1c4/frame 0xb70ec3c sleepq_switch(1819fadc,0) at sleepq_switch+0xf7/frame 0xb70ec5c _cv_wait(1819fadc,1819fcd8) at _cv_wait+0x135/frame 0xb70ec94 usb_process(1819fad4,b70ece8) at usb_process+0x9d/frame 0xb70ecb4 fork_exit(e3aa70,1819fad4,b70ece8,0,0,...) at fork_exit+0x69/frame 0xb70ecd4 fork_trampoline() at 0xffc033de/frame 0xb70ecd4 --- kthread start Tracing command usb pid 15 tid 100062 td 0x181a4180 sched_switch(181a4180,104) at sched_switch+0x4f3/frame 0xb713c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb713c3c sleepq_switch(1819fb0c,0) at sleepq_switch+0xf7/frame 0xb713c5c _cv_wait(1819fb0c,1819fcd8) at _cv_wait+0x135/frame 0xb713c94 usb_process(1819fb04,b713ce8) at usb_process+0x9d/frame 0xb713cb4 fork_exit(e3aa70,1819fb04,b713ce8,0,0,...) at fork_exit+0x69/frame 0xb713cd4 fork_trampoline() at 0xffc033de/frame 0xb713cd4 --- kthread start Tracing command usb pid 15 tid 100063 td 0x181a3e00 sched_switch(181a3e00,104) at sched_switch+0x4f3/frame 0xb718c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb718c3c sleepq_switch(1819fb3c,0) at sleepq_switch+0xf7/frame 0xb718c5c _cv_wait(1819fb3c,1819fcd8) at _cv_wait+0x135/frame 0xb718c94 usb_process(1819fb34,b718ce8) at usb_process+0x9d/frame 0xb718cb4 fork_exit(e3aa70,1819fb34,b718ce8,0,0,...) at fork_exit+0x69/frame 0xb718cd4 fork_trampoline() at 0xffc033de/frame 0xb718cd4 --- kthread start Tracing command usb pid 15 tid 100064 td 0x181a3a80 sched_switch(181a3a80,104) at sched_switch+0x4f3/frame 0xb71dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb71dc3c sleepq_switch(1819fb6c,0) at sleepq_switch+0xf7/frame 0xb71dc5c _cv_wait(1819fb6c,1819fcd8) at _cv_wait+0x135/frame 0xb71dc94 usb_process(1819fb64,b71dce8) at usb_process+0x9d/frame 0xb71dcb4 fork_exit(e3aa70,1819fb64,b71dce8,0,0,...) at fork_exit+0x69/frame 0xb71dcd4 fork_trampoline() at 0xffc033de/frame 0xb71dcd4 --- kthread start Tracing command usb pid 15 tid 100065 td 0x181a3700 sched_switch(181a3700,104) at sched_switch+0x4f3/frame 0xb723c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb723c3c sleepq_switch(181a7aac,0) at sleepq_switch+0xf7/frame 0xb723c5c _cv_wait(181a7aac,181a7cd8) at _cv_wait+0x135/frame 0xb723c94 usb_process(181a7aa4,b723ce8) at usb_process+0x9d/frame 0xb723cb4 fork_exit(e3aa70,181a7aa4,b723ce8,0,0,...) at fork_exit+0x69/frame 0xb723cd4 fork_trampoline() at 0xffc033de/frame 0xb723cd4 --- kthread start Tracing command usb pid 15 tid 100066 td 0x181a3380 sched_switch(181a3380,104) at sched_switch+0x4f3/frame 0xb728c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb728c3c sleepq_switch(181a7adc,0) at sleepq_switch+0xf7/frame 0xb728c5c _cv_wait(181a7adc,181a7cd8) at _cv_wait+0x135/frame 0xb728c94 usb_process(181a7ad4,b728ce8) at usb_process+0x9d/frame 0xb728cb4 fork_exit(e3aa70,181a7ad4,b728ce8,0,0,...) at fork_exit+0x69/frame 0xb728cd4 fork_trampoline() at 0xffc033de/frame 0xb728cd4 --- kthread start Tracing command usb pid 15 tid 100067 td 0x181a3000 sched_switch(181a3000,104) at sched_switch+0x4f3/frame 0xb72dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb72dc3c sleepq_switch(181a7b0c,0) at sleepq_switch+0xf7/frame 0xb72dc5c _cv_wait(181a7b0c,181a7cd8) at _cv_wait+0x135/frame 0xb72dc94 usb_process(181a7b04,b72dce8) at usb_process+0x9d/frame 0xb72dcb4 fork_exit(e3aa70,181a7b04,b72dce8,0,0,...) at fork_exit+0x69/frame 0xb72dcd4 fork_trampoline() at 0xffc033de/frame 0xb72dcd4 --- kthread start Tracing command usb pid 15 tid 100068 td 0x17ee4c00 sched_switch(17ee4c00,104) at sched_switch+0x4f3/frame 0xb732c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb732c3c sleepq_switch(181a7b3c,0) at sleepq_switch+0xf7/frame 0xb732c5c _cv_wait(181a7b3c,181a7cd8) at _cv_wait+0x135/frame 0xb732c94 usb_process(181a7b34,b732ce8) at usb_process+0x9d/frame 0xb732cb4 fork_exit(e3aa70,181a7b34,b732ce8,0,0,...) at fork_exit+0x69/frame 0xb732cd4 fork_trampoline() at 0xffc033de/frame 0xb732cd4 --- kthread start Tracing command usb pid 15 tid 100069 td 0x17ee4880 sched_switch(17ee4880,104) at sched_switch+0x4f3/frame 0xb737c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb737c3c sleepq_switch(181a7b6c,0) at sleepq_switch+0xf7/frame 0xb737c5c _cv_wait(181a7b6c,181a7cd8) at _cv_wait+0x135/frame 0xb737c94 usb_process(181a7b64,b737ce8) at usb_process+0x9d/frame 0xb737cb4 fork_exit(e3aa70,181a7b64,b737ce8,0,0,...) at fork_exit+0x69/frame 0xb737cd4 fork_trampoline() at 0xffc033de/frame 0xb737cd4 --- kthread start Tracing command usb pid 15 tid 100070 td 0x17ee4500 sched_switch(17ee4500,104) at sched_switch+0x4f3/frame 0xb73dc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb73dc3c sleepq_switch(181abaac,0) at sleepq_switch+0xf7/frame 0xb73dc5c _cv_wait(181abaac,181abcd8) at _cv_wait+0x135/frame 0xb73dc94 usb_process(181abaa4,b73dce8) at usb_process+0x9d/frame 0xb73dcb4 fork_exit(e3aa70,181abaa4,b73dce8,0,0,...) at fork_exit+0x69/frame 0xb73dcd4 fork_trampoline() at 0xffc033de/frame 0xb73dcd4 --- kthread start Tracing command usb pid 15 tid 100071 td 0x17ee4180 sched_switch(17ee4180,104) at sched_switch+0x4f3/frame 0xb742c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb742c3c sleepq_switch(181abadc,0) at sleepq_switch+0xf7/frame 0xb742c5c _cv_wait(181abadc,181abcd8) at _cv_wait+0x135/frame 0xb742c94 usb_process(181abad4,b742ce8) at usb_process+0x9d/frame 0xb742cb4 fork_exit(e3aa70,181abad4,b742ce8,0,0,...) at fork_exit+0x69/frame 0xb742cd4 fork_trampoline() at 0xffc033de/frame 0xb742cd4 --- kthread start Tracing command usb pid 15 tid 100072 td 0x17ee3e00 sched_switch(17ee3e00,104) at sched_switch+0x4f3/frame 0xb747c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb747c3c sleepq_switch(181abb0c,0) at sleepq_switch+0xf7/frame 0xb747c5c _cv_wait(181abb0c,181abcd8) at _cv_wait+0x135/frame 0xb747c94 usb_process(181abb04,b747ce8) at usb_process+0x9d/frame 0xb747cb4 fork_exit(e3aa70,181abb04,b747ce8,0,0,...) at fork_exit+0x69/frame 0xb747cd4 fork_trampoline() at 0xffc033de/frame 0xb747cd4 --- kthread start Tracing command usb pid 15 tid 100073 td 0x17ee3a80 sched_switch(17ee3a80,104) at sched_switch+0x4f3/frame 0xb74cc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb74cc3c sleepq_switch(181abb3c,0) at sleepq_switch+0xf7/frame 0xb74cc5c _cv_wait(181abb3c,181abcd8) at _cv_wait+0x135/frame 0xb74cc94 usb_process(181abb34,b74cce8) at usb_process+0x9d/frame 0xb74ccb4 fork_exit(e3aa70,181abb34,b74cce8,0,0,...) at fork_exit+0x69/frame 0xb74ccd4 fork_trampoline() at 0xffc033de/frame 0xb74ccd4 --- kthread start Tracing command usb pid 15 tid 100074 td 0x17ee3700 sched_switch(17ee3700,104) at sched_switch+0x4f3/frame 0xb751c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb751c3c sleepq_switch(181abb6c,0) at sleepq_switch+0xf7/frame 0xb751c5c _cv_wait(181abb6c,181abcd8) at _cv_wait+0x135/frame 0xb751c94 usb_process(181abb64,b751ce8) at usb_process+0x9d/frame 0xb751cb4 fork_exit(e3aa70,181abb64,b751ce8,0,0,...) at fork_exit+0x69/frame 0xb751cd4 fork_trampoline() at 0xffc033de/frame 0xb751cd4 --- kthread start Tracing command usb pid 15 tid 100076 td 0x181cd180 sched_switch(181cd180,104) at sched_switch+0x4f3/frame 0xb75cc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb75cc3c sleepq_switch(17ebdc84,0) at sleepq_switch+0xf7/frame 0xb75cc5c _cv_wait(17ebdc84,17ebdeb0) at _cv_wait+0x135/frame 0xb75cc94 usb_process(17ebdc7c,b75cce8) at usb_process+0x9d/frame 0xb75ccb4 fork_exit(e3aa70,17ebdc7c,b75cce8,0,0,...) at fork_exit+0x69/frame 0xb75ccd4 fork_trampoline() at 0xffc033de/frame 0xb75ccd4 --- kthread start Tracing command usb pid 15 tid 100077 td 0x181cce00 sched_switch(181cce00,104) at sched_switch+0x4f3/frame 0xb761c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb761c3c sleepq_switch(17ebdcb4,0) at sleepq_switch+0xf7/frame 0xb761c5c _cv_wait(17ebdcb4,17ebdeb0) at _cv_wait+0x135/frame 0xb761c94 usb_process(17ebdcac,b761ce8) at usb_process+0x9d/frame 0xb761cb4 fork_exit(e3aa70,17ebdcac,b761ce8,0,0,...) at fork_exit+0x69/frame 0xb761cd4 fork_trampoline() at 0xffc033de/frame 0xb761cd4 --- kthread start Tracing command usb pid 15 tid 100078 td 0x181cca80 sched_switch(181cca80,104) at sched_switch+0x4f3/frame 0xb766c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb766c3c sleepq_switch(17ebdce4,0) at sleepq_switch+0xf7/frame 0xb766c5c _cv_wait(17ebdce4,17ebdeb0) at _cv_wait+0x135/frame 0xb766c94 usb_process(17ebdcdc,b766ce8) at usb_process+0x9d/frame 0xb766cb4 fork_exit(e3aa70,17ebdcdc,b766ce8,0,0,...) at fork_exit+0x69/frame 0xb766cd4 fork_trampoline() at 0xffc033de/frame 0xb766cd4 --- kthread start Tracing command usb pid 15 tid 100079 td 0x181cc700 sched_switch(181cc700,104) at sched_switch+0x4f3/frame 0xb76bc00 mi_switch(104) at mi_switch+0x1c4/frame 0xb76bc3c sleepq_switch(17ebdd14,0) at sleepq_switch+0xf7/frame 0xb76bc5c _cv_wait(17ebdd14,17ebdeb0) at _cv_wait+0x135/frame 0xb76bc94 usb_process(17ebdd0c,b76bce8) at usb_process+0x9d/frame 0xb76bcb4 fork_exit(e3aa70,17ebdd0c,b76bce8,0,0,...) at fork_exit+0x69/frame 0xb76bcd4 fork_trampoline() at 0xffc033de/frame 0xb76bcd4 --- kthread start Tracing command usb pid 15 tid 100080 td 0x181cc380 sched_switch(181cc380,104) at sched_switch+0x4f3/frame 0xb770c00 mi_switch(104) at mi_switch+0x1c4/frame 0xb770c3c sleepq_switch(17ebdd44,0) at sleepq_switch+0xf7/frame 0xb770c5c _cv_wait(17ebdd44,17ebdeb0) at _cv_wait+0x135/frame 0xb770c94 usb_process(17ebdd3c,b770ce8) at usb_process+0x9d/frame 0xb770cb4 fork_exit(e3aa70,17ebdd3c,b770ce8,0,0,...) at fork_exit+0x69/frame 0xb770cd4 fork_trampoline() at 0xffc033de/frame 0xb770cd4 --- kthread start Tracing command cam pid 7 tid 100043 td 0x9985380 (CPU 2) cpustop_handler(46,b6849c0,b6849b4,1417ccb,f,...) at cpustop_handler+0x24/frame 0xb6848e8 ipi_nmi_handler(f,1,54,7fffffff,9deed50,...) at ipi_nmi_handler+0x31/frame 0xb6848f8 trap(b6849c0,8,28,28,1e4c9d0,...) at trap+0xbb/frame 0xb6849b4 calltrap() at 0xffc0316f/frame 0xb6849b4 --- trap 0x13, eip = 0x10364b0, esp = 0xb684a00, ebp = 0xb684a20 --- witness_unlock(1e4c9d0,0,155a962,ce2) at witness_unlock/frame 0xb684a20 vm_page_pqstate_commit(3488310,b684a88,248) at vm_page_pqstate_commit+0x1e0/frame 0xb684a74 vm_page_deactivate_noreuse(3488310,2674d458,2674d3e8,2674d3d8,18cd7380,...) at vm_page_deactivate_noreuse+0x64/frame 0xb684aa0 swp_pager_async_iodone(18cd7380) at swp_pager_async_iodone+0x18d/frame 0xb684ac4 bufdone(18cd7380) at bufdone+0x3e/frame 0xb684b10 swapgeom_done(2342a450) at swapgeom_done+0x43/frame 0xb684b24 g_io_deliver(2342a450,0) at g_io_deliver+0x247/frame 0xb684b60 g_std_done(2342a564) at g_std_done+0x78/frame 0xb684b74 g_io_deliver(2342a564,0) at g_io_deliver+0x247/frame 0xb684bb0 g_std_done(2342a78c) at g_std_done+0x78/frame 0xb684bc4 g_io_deliver(2342a78c,0) at g_io_deliver+0x247/frame 0xb684c00 g_disk_done(2342aac8) at g_disk_done+0xf0/frame 0xb684c34 adadone(9de5700,99c2800) at adadone+0xc6d/frame 0xb684c6c xpt_done_process(0,b684c98,1b2f510,202,9985380,...) at xpt_done_process+0x2f7/frame 0xb684c90 xpt_done_td(1b2f500,b684ce8) at xpt_done_td+0xd5/frame 0xb684cb4 fork_exit(907280,1b2f500,b684ce8,0,0,...) at fork_exit+0x69/frame 0xb684cd4 fork_trampoline() at 0xffc033de/frame 0xb684cd4 --- kthread start Tracing command cam pid 7 tid 100096 td 0x18863500 sched_switch(18863500,104) at sched_switch+0x4f3/frame 0xb7c9bc8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7c9c04 sleepq_switch(1b2f468,5c) at sleepq_switch+0xf7/frame 0xb7c9c24 _sleep(1b2f468,1b2f4d8,5c,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7c9c64 xpt_scanner_thread(0,b7c9ce8) at xpt_scanner_thread+0x56/frame 0xb7c9cb4 fork_exit(907520,0,b7c9ce8,0,0,...) at fork_exit+0x69/frame 0xb7c9cd4 fork_trampoline() at 0xffc033de/frame 0xb7c9cd4 --- kthread start Tracing command crypto returns 3 pid 6 tid 100042 td 0x9985700 sched_switch(9985700,104) at sched_switch+0x4f3/frame 0xb67fbc8 mi_switch(104) at mi_switch+0x1c4/frame 0xb67fc04 sleepq_switch(9906dc4,6c) at sleepq_switch+0xf7/frame 0xb67fc24 _sleep(9906dc4,9906da8,6c,155a20b,0,0,0,0,100) at _sleep+0x2f7/frame 0xb67fc64 crypto_ret_proc(9906da8,b67fce8) at crypto_ret_proc+0x202/frame 0xb67fcb4 fork_exit(12427a0,9906da8,b67fce8,0,0,...) at fork_exit+0x69/frame 0xb67fcd4 fork_trampoline() at 0xffc033de/frame 0xb67fcd4 --- kthread start Tracing command crypto returns 2 pid 5 tid 100041 td 0x9985a80 sched_switch(9985a80,104) at sched_switch+0x4f3/frame 0x129fcbc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129fcc04 sleepq_switch(9906d8c,6c) at sleepq_switch+0xf7/frame 0x129fcc24 _sleep(9906d8c,9906d70,6c,155a20b,0,0,0,0,100) at _sleep+0x2f7/frame 0x129fcc64 crypto_ret_proc(9906d70,129fcce8) at crypto_ret_proc+0x202/frame 0x129fccb4 fork_exit(12427a0,9906d70,129fcce8,0,0,...) at fork_exit+0x69/frame 0x129fccd4 fork_trampoline() at 0xffc033de/frame 0x129fccd4 --- kthread start Tracing command crypto returns 1 pid 4 tid 100040 td 0x9985e00 sched_switch(9985e00,104) at sched_switch+0x4f3/frame 0x129f7bc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129f7c04 sleepq_switch(9906d54,6c) at sleepq_switch+0xf7/frame 0x129f7c24 _sleep(9906d54,9906d38,6c,155a20b,0,0,0,0,100) at _sleep+0x2f7/frame 0x129f7c64 crypto_ret_proc(9906d38,129f7ce8) at crypto_ret_proc+0x202/frame 0x129f7cb4 fork_exit(12427a0,9906d38,129f7ce8,0,0,...) at fork_exit+0x69/frame 0x129f7cd4 fork_trampoline() at 0xffc033de/frame 0x129f7cd4 --- kthread start Tracing command crypto returns 0 pid 3 tid 100039 td 0x9986180 sched_switch(9986180,104) at sched_switch+0x4f3/frame 0x129f2bc8 mi_switch(104) at mi_switch+0x1c4/frame 0x129f2c04 sleepq_switch(9906d1c,6c) at sleepq_switch+0xf7/frame 0x129f2c24 _sleep(9906d1c,9906d00,6c,155a20b,0,0,0,0,100) at _sleep+0x2f7/frame 0x129f2c64 crypto_ret_proc(9906d00,129f2ce8) at crypto_ret_proc+0x202/frame 0x129f2cb4 fork_exit(12427a0,9906d00,129f2ce8,0,0,...) at fork_exit+0x69/frame 0x129f2cd4 fork_trampoline() at 0xffc033de/frame 0x129f2cd4 --- kthread start Tracing command crypto pid 2 tid 100038 td 0x9986500 sched_switch(9986500,104) at sched_switch+0x4f3/frame 0x129edbd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129edc14 sleepq_switch(1e12b28,6c) at sleepq_switch+0xf7/frame 0x129edc34 _sleep(1e12b28,1e12b10,6c,150d780,0,0,0,0,100) at _sleep+0x2f7/frame 0x129edc74 crypto_proc(0,129edce8) at crypto_proc+0x258/frame 0x129edcb4 fork_exit(1242400,0,129edce8,0,0,...) at fork_exit+0x69/frame 0x129edcd4 fork_trampoline() at 0xffc033de/frame 0x129edcd4 --- kthread start Tracing command sequencer 00 pid 14 tid 100033 td 0x9987700 sched_switch(9987700,104) at sched_switch+0x4f3/frame 0x129d4bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129d4c14 sleepq_switch(9d1564c,0) at sleepq_switch+0xf7/frame 0x129d4c34 _cv_wait(9d1564c,9d15604) at _cv_wait+0x135/frame 0x129d4c6c seq_eventthread(9d15600,129d4ce8) at seq_eventthread+0x7b/frame 0x129d4cb4 fork_exit(dc7c50,9d15600,129d4ce8,0,0,...) at fork_exit+0x69/frame 0x129d4cd4 fork_trampoline() at 0xffc033de/frame 0x129d4cd4 --- kthread start Tracing command geom pid 13 tid 100030 td 0x991b000 sched_switch(991b000,104) at sched_switch+0x4f3/frame 0x129c5be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129c5c1c sleepq_switch(1e4c2f0,5c) at sleepq_switch+0xf7/frame 0x129c5c3c _sleep(1e4c2f0,1d8c768,25c,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129c5c7c g_run_events(0,129c5ce8) at g_run_events+0x4d/frame 0x129c5cb4 fork_exit(f2c910,0,129c5ce8,0,0,...) at fork_exit+0x69/frame 0x129c5cd4 fork_trampoline() at 0xffc033de/frame 0x129c5cd4 --- kthread start Tracing command geom pid 13 tid 100031 td 0x9df9c00 sched_switch(9df9c00,104) at sched_switch+0x4f3/frame 0x129cabd8 mi_switch(104) at mi_switch+0x1c4/frame 0x129cac14 sleepq_switch(1e4c2f8,5c) at sleepq_switch+0xf7/frame 0x129cac34 _sleep(1e4c2f8,1d8c7ac,25c,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129cac74 g_io_schedule_up(9df9c00) at g_io_schedule_up+0x36/frame 0x129caca8 g_up_procbody(0,129cace8) at g_up_procbody+0x5b/frame 0x129cacb4 fork_exit(f2c960,0,129cace8,0,0,...) at fork_exit+0x69/frame 0x129cacd4 fork_trampoline() at 0xffc033de/frame 0x129cacd4 --- kthread start Tracing command geom pid 13 tid 100032 td 0x9df9880 sched_switch(9df9880,104) at sched_switch+0x4f3/frame 0x129cfbd0 mi_switch(104) at mi_switch+0x1c4/frame 0x129cfc10 sleepq_switch(1e4c2f4,5c) at sleepq_switch+0xf7/frame 0x129cfc30 _sleep(1e4c2f4,1d8c78c,25c,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129cfc70 g_io_schedule_down(9df9880) at g_io_schedule_down+0x36/frame 0x129cfca8 g_down_procbody(0,129cfce8) at g_down_procbody+0x5b/frame 0x129cfcb4 fork_exit(f2c9c0,0,129cfce8,0,0,...) at fork_exit+0x69/frame 0x129cfcd4 fork_trampoline() at 0xffc033de/frame 0x129cfcd4 --- kthread start Tracing command intr pid 12 tid 100007 td 0x9df8000 sched_switch(9df8000,109) at sched_switch+0x4f3/frame 0x1293ac28 mi_switch(109) at mi_switch+0x1c4/frame 0x1293ac64 ithread_loop(988efa0,1293ace8) at ithread_loop+0x85/frame 0x1293acb4 fork_exit(f9b620,988efa0,1293ace8,0,0,...) at fork_exit+0x69/frame 0x1293acd4 fork_trampoline() at 0xffc033de/frame 0x1293acd4 --- kthread start Tracing command intr pid 12 tid 100009 td 0x9df3880 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100013 td 0x9df2a80 sched_switch(9df2a80,109) at sched_switch+0x4f3/frame 0x12958c28 mi_switch(109) at mi_switch+0x1c4/frame 0x12958c64 ithread_loop(988ef80,12958ce8) at ithread_loop+0x85/frame 0x12958cb4 fork_exit(f9b620,988ef80,12958ce8,0,0,...) at fork_exit+0x69/frame 0x12958cd4 fork_trampoline() at 0xffc033de/frame 0x12958cd4 --- kthread start Tracing command intr pid 12 tid 100023 td 0x991c880 sched_switch(991c880,109) at sched_switch+0x4f3/frame 0x12996c28 mi_switch(109) at mi_switch+0x1c4/frame 0x12996c64 ithread_loop(988ef70,12996ce8) at ithread_loop+0x85/frame 0x12996cb4 fork_exit(f9b620,988ef70,12996ce8,0,0,...) at fork_exit+0x69/frame 0x12996cd4 fork_trampoline() at 0xffc033de/frame 0x12996cd4 --- kthread start Tracing command intr pid 12 tid 100024 td 0x991c500 sched_switch(991c500,109) at sched_switch+0x4f3/frame 0x1299bc28 mi_switch(109) at mi_switch+0x1c4/frame 0x1299bc64 ithread_loop(988ef60,1299bce8) at ithread_loop+0x85/frame 0x1299bcb4 fork_exit(f9b620,988ef60,1299bce8,0,0,...) at fork_exit+0x69/frame 0x1299bcd4 fork_trampoline() at 0xffc033de/frame 0x1299bcd4 --- kthread start Tracing command intr pid 12 tid 100025 td 0x991c180 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100026 td 0x991be00 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100027 td 0x991ba80 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100028 td 0x991b700 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100044 td 0x17ee3000 sched_switch(17ee3000,109) at sched_switch+0x4f3/frame 0xb6b4c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6b4c64 ithread_loop(17ed4030,b6b4ce8) at ithread_loop+0x85/frame 0xb6b4cb4 fork_exit(f9b620,17ed4030,b6b4ce8,0,0,...) at fork_exit+0x69/frame 0xb6b4cd4 fork_trampoline() at 0xffc033de/frame 0xb6b4cd4 --- kthread start Tracing command intr pid 12 tid 100045 td 0x17ed6c00 sched_switch(17ed6c00,109) at sched_switch+0x4f3/frame 0xb6bac28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6bac64 ithread_loop(17ed4020,b6bace8) at ithread_loop+0x85/frame 0xb6bacb4 fork_exit(f9b620,17ed4020,b6bace8,0,0,...) at fork_exit+0x69/frame 0xb6bacd4 fork_trampoline() at 0xffc033de/frame 0xb6bacd4 --- kthread start Tracing command intr pid 12 tid 100046 td 0x17ed6880 sched_switch(17ed6880,109) at sched_switch+0x4f3/frame 0xb6c0c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6c0c64 ithread_loop(17ed4010,b6c0ce8) at ithread_loop+0x85/frame 0xb6c0cb4 fork_exit(f9b620,17ed4010,b6c0ce8,0,0,...) at fork_exit+0x69/frame 0xb6c0cd4 fork_trampoline() at 0xffc033de/frame 0xb6c0cd4 --- kthread start Tracing command intr pid 12 tid 100047 td 0x17ed6500 sched_switch(17ed6500,109) at sched_switch+0x4f3/frame 0xb6c6c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6c6c64 ithread_loop(17ed4000,b6c6ce8) at ithread_loop+0x85/frame 0xb6c6cb4 fork_exit(f9b620,17ed4000,b6c6ce8,0,0,...) at fork_exit+0x69/frame 0xb6c6cd4 fork_trampoline() at 0xffc033de/frame 0xb6c6cd4 --- kthread start Tracing command intr pid 12 tid 100053 td 0x17ed5000 sched_switch(17ed5000,109) at sched_switch+0x4f3/frame 0xb6e5c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb6e5c64 ithread_loop(988ffa0,b6e5ce8) at ithread_loop+0x85/frame 0xb6e5cb4 fork_exit(f9b620,988ffa0,b6e5ce8,0,0,...) at fork_exit+0x69/frame 0xb6e5cd4 fork_trampoline() at 0xffc033de/frame 0xb6e5cd4 --- kthread start Tracing command intr pid 12 tid 100059 td 0x9987a80 sched_switch(9987a80,109) at sched_switch+0x4f3/frame 0xb704c28 mi_switch(109) at mi_switch+0x1c4/frame 0xb704c64 ithread_loop(988ff90,b704ce8) at ithread_loop+0x85/frame 0xb704cb4 fork_exit(f9b620,988ff90,b704ce8,0,0,...) at fork_exit+0x69/frame 0xb704cd4 fork_trampoline() at 0xffc033de/frame 0xb704cd4 --- kthread start Tracing command intr pid 12 tid 100075 td 0x17ee3380 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100081 td 0x181cc000 exit_tag() at 0xffc033d4 Tracing command intr pid 12 tid 100082 td 0x181cbc00 sched_switch(181cbc00,109) at sched_switch+0x4f3/frame 0xb77ec28 mi_switch(109) at mi_switch+0x1c4/frame 0xb77ec64 ithread_loop(988ff60,b77ece8) at ithread_loop+0x85/frame 0xb77ecb4 fork_exit(f9b620,988ff60,b77ece8,0,0,...) at fork_exit+0x69/frame 0xb77ecd4 fork_trampoline() at 0xffc033de/frame 0xb77ecd4 --- kthread start Tracing command intr pid 12 tid 100084 td 0x181cb500 exit_tag() at 0xffc033d4 Tracing command idle pid 11 tid 100003 td 0x9df8e00 sched_switch(9df8e00,608) at sched_switch+0x4f3/frame 0x12925c10 mi_switch(608) at mi_switch+0x1c4/frame 0x12925c4c critical_exit_preempt(0) at critical_exit_preempt+0x5c/frame 0x12925c5c sched_idletd(0,12925ce8) at sched_idletd+0x32e/frame 0x12925cb4 fork_exit(10007f0,0,12925ce8,0,0,...) at fork_exit+0x69/frame 0x12925cd4 fork_trampoline() at 0xffc033de/frame 0x12925cd4 --- kthread start Tracing command idle pid 11 tid 100004 td 0x9df8a80 sched_switch(9df8a80,108) at sched_switch+0x4f3/frame 0x1292ac20 mi_switch(108) at mi_switch+0x1c4/frame 0x1292ac5c sched_idletd(0,1292ace8) at sched_idletd+0x38d/frame 0x1292acb4 fork_exit(10007f0,0,1292ace8,0,0,...) at fork_exit+0x69/frame 0x1292acd4 fork_trampoline() at 0xffc033de/frame 0x1292acd4 --- kthread start Tracing command idle pid 11 tid 100005 td 0x9df8700 sched_switch(9df8700,108) at sched_switch+0x4f3/frame 0x1292fc20 mi_switch(108) at mi_switch+0x1c4/frame 0x1292fc5c sched_idletd(0,1292fce8) at sched_idletd+0x38d/frame 0x1292fcb4 fork_exit(10007f0,0,1292fce8,0,0,...) at fork_exit+0x69/frame 0x1292fcd4 fork_trampoline() at 0xffc033de/frame 0x1292fcd4 --- kthread start Tracing command idle pid 11 tid 100006 td 0x9df8380 sched_switch(9df8380,108) at sched_switch+0x4f3/frame 0x12934c20 mi_switch(108) at mi_switch+0x1c4/frame 0x12934c5c sched_idletd(0,12934ce8) at sched_idletd+0x38d/frame 0x12934cb4 fork_exit(10007f0,0,12934ce8,0,0,...) at fork_exit+0x69/frame 0x12934cd4 fork_trampoline() at 0xffc033de/frame 0x12934cd4 --- kthread start Tracing command init pid 1 tid 100002 td 0x9df9180 sched_switch(9df9180,104) at sched_switch+0x4f3/frame 0x1291c9b8 mi_switch(104) at mi_switch+0x1c4/frame 0x1291c9f8 sleepq_switch(2d00,6c,0,1da2bd4,9df10a4,...) at sleepq_switch+0xf7/frame 0x1291ca18 sleepq_catch_signals(1291ca8c,fdf10e,9df10a4,6c,200246,...) at sleepq_catch_signals+0x368/frame 0x1291ca44 sleepq_wait_sig(9df10a4,6c) at sleepq_wait_sig+0xe/frame 0x1291ca4c _sleep(9df10a4,9df114c,36c,1544874,0,0,0,0,100) at _sleep+0x27e/frame 0x1291ca8c kern_wait6(9df9180,7,0,0,1291cb14,30,0,0) at kern_wait6+0x3b8/frame 0x1291cae8 sys_wait4(9df9180,9df940c) at sys_wait4+0x79/frame 0x1291cc08 syscall(1291cce8,3b,3b,3b,ffbfe918,...) at syscall+0x306/frame 0x1291ccdc Xint0x80_syscall() at 0xffc033c9/frame 0x1291ccdc --- syscall (0, FreeBSD ELF32, nosys), eip = 0x4026f0, esp = 0xffbfe638, ebp = 0xffbfe650 --- Tracing command audit pid 10 tid 100001 td 0x9df9500 sched_switch(9df9500,104) at sched_switch+0x4f3/frame 0x12917bd8 mi_switch(104) at mi_switch+0x1c4/frame 0x12917c18 sleepq_switch(1e5e50c,0) at sleepq_switch+0xf7/frame 0x12917c38 _cv_wait(1e5e50c,1e5e4b8) at _cv_wait+0x135/frame 0x12917c70 audit_worker(0,12917ce8) at audit_worker+0x88/frame 0x12917cb4 fork_exit(12619c0,0,12917ce8,0,0,...) at fork_exit+0x69/frame 0x12917cd4 fork_trampoline() at 0xffc033de/frame 0x12917cd4 --- kthread start Tracing command kernel pid 0 tid 100000 td 0x1e4c380 sched_switch(1e4c380,104) at sched_switch+0x4f3/frame 0x22f9c08 mi_switch(104) at mi_switch+0x1c4/frame 0x22f9c48 sleepq_switch(1e4c380,0,1e21ce0,22f9ccc,1e4c380,...) at sleepq_switch+0xf7/frame 0x22f9c68 sleepq_timedwait(1e21ce0,54,246,0,40000,...) at sleepq_timedwait+0x3f/frame 0x22f9c8c _sleep(1e21ce0,0,54,15476f2,fffff470,9,0,0,100) at _sleep+0x22c/frame 0x22f9ccc swapper(0,0,0,1b124fc) at swapper+0x6d/frame 0x22f9d14 mi_startup() at mi_startup+0x2c6/frame 0x22f9d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100008 td 0x9df3c00 sched_switch(9df3c00,104) at sched_switch+0x4f3/frame 0x1293fbe0 mi_switch(104) at mi_switch+0x1c4/frame 0x1293fc1c sleepq_switch(9d3b600,0) at sleepq_switch+0xf7/frame 0x1293fc3c _sleep(9d3b600,9d3b640,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x1293fc7c taskqueue_thread_loop(1e214cc,1293fce8) at taskqueue_thread_loop+0x9b/frame 0x1293fcb4 fork_exit(102ad60,1e214cc,1293fce8,0,0,...) at fork_exit+0x69/frame 0x1293fcd4 fork_trampoline() at 0xffc033de/frame 0x1293fcd4 --- kthread start Tracing command kernel pid 0 tid 100010 td 0x9df3500 sched_switch(9df3500,104) at sched_switch+0x4f3/frame 0x12949be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12949c28 sleepq_switch(9d36e00,0) at sleepq_switch+0xf7/frame 0x12949c48 msleep_spin_sbt(9d36e00,9d36e40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12949c80 gtaskqueue_thread_loop(990dc04,12949ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12949cb4 fork_exit(10167e0,990dc04,12949ce8,0,0,...) at fork_exit+0x69/frame 0x12949cd4 fork_trampoline() at 0xffc033de/frame 0x12949cd4 --- kthread start Tracing command kernel pid 0 tid 100011 td 0x9df3180 sched_switch(9df3180,104) at sched_switch+0x4f3/frame 0x1294ebe0 mi_switch(104) at mi_switch+0x1c4/frame 0x1294ec1c sleepq_switch(9d3b100,0) at sleepq_switch+0xf7/frame 0x1294ec3c _sleep(9d3b100,9d3b140,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x1294ec7c taskqueue_thread_loop(1e4cff0,1294ece8) at taskqueue_thread_loop+0x9b/frame 0x1294ecb4 fork_exit(102ad60,1e4cff0,1294ece8,0,0,...) at fork_exit+0x69/frame 0x1294ecd4 fork_trampoline() at 0xffc033de/frame 0x1294ecd4 --- kthread start Tracing command kernel pid 0 tid 100012 td 0x9df2e00 sched_switch(9df2e00,104) at sched_switch+0x4f3/frame 0x12953be0 mi_switch(104) at mi_switch+0x1c4/frame 0x12953c1c sleepq_switch(9d3ae00,0) at sleepq_switch+0xf7/frame 0x12953c3c _sleep(9d3ae00,9d3ae40,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x12953c7c taskqueue_thread_loop(1e5d940,12953ce8) at taskqueue_thread_loop+0x9b/frame 0x12953cb4 fork_exit(102ad60,1e5d940,12953ce8,0,0,...) at fork_exit+0x69/frame 0x12953cd4 fork_trampoline() at 0xffc033de/frame 0x12953cd4 --- kthread start Tracing command kernel pid 0 tid 100014 td 0x9df2700 sched_switch(9df2700,104) at sched_switch+0x4f3/frame 0x12969be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12969c28 sleepq_switch(9d36d00,0) at sleepq_switch+0xf7/frame 0x12969c48 msleep_spin_sbt(9d36d00,9d36d40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12969c80 gtaskqueue_thread_loop(990d004,12969ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12969cb4 fork_exit(10167e0,990d004,12969ce8,0,0,...) at fork_exit+0x69/frame 0x12969cd4 fork_trampoline() at 0xffc033de/frame 0x12969cd4 --- kthread start Tracing command kernel pid 0 tid 100015 td 0x9df2380 sched_switch(9df2380,104) at sched_switch+0x4f3/frame 0x1296ebe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1296ec28 sleepq_switch(9d36c00,0) at sleepq_switch+0xf7/frame 0x1296ec48 msleep_spin_sbt(9d36c00,9d36c40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1296ec80 gtaskqueue_thread_loop(990d014,1296ece8) at gtaskqueue_thread_loop+0xd9/frame 0x1296ecb4 fork_exit(10167e0,990d014,1296ece8,0,0,...) at fork_exit+0x69/frame 0x1296ecd4 fork_trampoline() at 0xffc033de/frame 0x1296ecd4 --- kthread start Tracing command kernel pid 0 tid 100016 td 0x9df2000 sched_switch(9df2000,104) at sched_switch+0x4f3/frame 0x12973be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12973c28 sleepq_switch(9d36b00,0) at sleepq_switch+0xf7/frame 0x12973c48 msleep_spin_sbt(9d36b00,9d36b40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12973c80 gtaskqueue_thread_loop(990d024,12973ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12973cb4 fork_exit(10167e0,990d024,12973ce8,0,0,...) at fork_exit+0x69/frame 0x12973cd4 fork_trampoline() at 0xffc033de/frame 0x12973cd4 --- kthread start Tracing command kernel pid 0 tid 100017 td 0x991de00 sched_switch(991de00,104) at sched_switch+0x4f3/frame 0x12978be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12978c28 sleepq_switch(9d36a00,0) at sleepq_switch+0xf7/frame 0x12978c48 msleep_spin_sbt(9d36a00,9d36a40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12978c80 gtaskqueue_thread_loop(990d034,12978ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12978cb4 fork_exit(10167e0,990d034,12978ce8,0,0,...) at fork_exit+0x69/frame 0x12978cd4 fork_trampoline() at 0xffc033de/frame 0x12978cd4 --- kthread start Tracing command kernel pid 0 tid 100018 td 0x991da80 sched_switch(991da80,104) at sched_switch+0x4f3/frame 0x1297dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1297dc28 sleepq_switch(9d36900,0) at sleepq_switch+0xf7/frame 0x1297dc48 msleep_spin_sbt(9d36900,9d36940,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1297dc80 gtaskqueue_thread_loop(990d804,1297dce8) at gtaskqueue_thread_loop+0xd9/frame 0x1297dcb4 fork_exit(10167e0,990d804,1297dce8,0,0,...) at fork_exit+0x69/frame 0x1297dcd4 fork_trampoline() at 0xffc033de/frame 0x1297dcd4 --- kthread start Tracing command kernel pid 0 tid 100019 td 0x991d700 sched_switch(991d700,104) at sched_switch+0x4f3/frame 0x12982be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12982c28 sleepq_switch(9d36800,0) at sleepq_switch+0xf7/frame 0x12982c48 msleep_spin_sbt(9d36800,9d36840,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12982c80 gtaskqueue_thread_loop(990d814,12982ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12982cb4 fork_exit(10167e0,990d814,12982ce8,0,0,...) at fork_exit+0x69/frame 0x12982cd4 fork_trampoline() at 0xffc033de/frame 0x12982cd4 --- kthread start Tracing command kernel pid 0 tid 100020 td 0x991d380 sched_switch(991d380,104) at sched_switch+0x4f3/frame 0x12987be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12987c28 sleepq_switch(9d36700,0) at sleepq_switch+0xf7/frame 0x12987c48 msleep_spin_sbt(9d36700,9d36740,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12987c80 gtaskqueue_thread_loop(990d824,12987ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12987cb4 fork_exit(10167e0,990d824,12987ce8,0,0,...) at fork_exit+0x69/frame 0x12987cd4 fork_trampoline() at 0xffc033de/frame 0x12987cd4 --- kthread start Tracing command kernel pid 0 tid 100021 td 0x991d000 sched_switch(991d000,104) at sched_switch+0x4f3/frame 0x1298cbe8 mi_switch(104) at mi_switch+0x1c4/frame 0x1298cc28 sleepq_switch(9d36600,0) at sleepq_switch+0xf7/frame 0x1298cc48 msleep_spin_sbt(9d36600,9d36640,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x1298cc80 gtaskqueue_thread_loop(990d834,1298cce8) at gtaskqueue_thread_loop+0xd9/frame 0x1298ccb4 fork_exit(10167e0,990d834,1298cce8,0,0,...) at fork_exit+0x69/frame 0x1298ccd4 fork_trampoline() at 0xffc033de/frame 0x1298ccd4 --- kthread start Tracing command kernel pid 0 tid 100022 td 0x991cc00 sched_switch(991cc00,104) at sched_switch+0x4f3/frame 0x12991be8 mi_switch(104) at mi_switch+0x1c4/frame 0x12991c28 sleepq_switch(9d36500,0) at sleepq_switch+0xf7/frame 0x12991c48 msleep_spin_sbt(9d36500,9d36540,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0x12991c80 gtaskqueue_thread_loop(990d404,12991ce8) at gtaskqueue_thread_loop+0xd9/frame 0x12991cb4 fork_exit(10167e0,990d404,12991ce8,0,0,...) at fork_exit+0x69/frame 0x12991cd4 fork_trampoline() at 0xffc033de/frame 0x12991cd4 --- kthread start Tracing command kernel pid 0 tid 100029 td 0x991b380 sched_switch(991b380,104) at sched_switch+0x4f3/frame 0x129c0be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129c0c1c sleepq_switch(9d39a00,0) at sleepq_switch+0xf7/frame 0x129c0c3c _sleep(9d39a00,9d39a40,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129c0c7c taskqueue_thread_loop(1d9f554,129c0ce8) at taskqueue_thread_loop+0x9b/frame 0x129c0cb4 fork_exit(102ad60,1d9f554,129c0ce8,0,0,...) at fork_exit+0x69/frame 0x129c0cd4 fork_trampoline() at 0xffc033de/frame 0x129c0cd4 --- kthread start Tracing command kernel pid 0 tid 100034 td 0x9987380 sched_switch(9987380,104) at sched_switch+0x4f3/frame 0x129d9be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129d9c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129d9c3c _sleep(9d39400,9d39440,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129d9c7c taskqueue_thread_loop(1e12bcc,129d9ce8) at taskqueue_thread_loop+0x9b/frame 0x129d9cb4 fork_exit(102ad60,1e12bcc,129d9ce8,0,0,...) at fork_exit+0x69/frame 0x129d9cd4 fork_trampoline() at 0xffc033de/frame 0x129d9cd4 --- kthread start Tracing command kernel pid 0 tid 100035 td 0x9987000 sched_switch(9987000,104) at sched_switch+0x4f3/frame 0x129debe0 mi_switch(104) at mi_switch+0x1c4/frame 0x129dec1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129dec3c _sleep(9d39400,9d39440,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129dec7c taskqueue_thread_loop(1e12bcc,129dece8) at taskqueue_thread_loop+0x9b/frame 0x129decb4 fork_exit(102ad60,1e12bcc,129dece8,0,0,...) at fork_exit+0x69/frame 0x129decd4 fork_trampoline() at 0xffc033de/frame 0x129decd4 --- kthread start Tracing command kernel pid 0 tid 100036 td 0x9986c00 sched_switch(9986c00,104) at sched_switch+0x4f3/frame 0x129e3be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129e3c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129e3c3c _sleep(9d39400,9d39440,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129e3c7c taskqueue_thread_loop(1e12bcc,129e3ce8) at taskqueue_thread_loop+0x9b/frame 0x129e3cb4 fork_exit(102ad60,1e12bcc,129e3ce8,0,0,...) at fork_exit+0x69/frame 0x129e3cd4 fork_trampoline() at 0xffc033de/frame 0x129e3cd4 --- kthread start Tracing command kernel pid 0 tid 100037 td 0x9986880 sched_switch(9986880,104) at sched_switch+0x4f3/frame 0x129e8be0 mi_switch(104) at mi_switch+0x1c4/frame 0x129e8c1c sleepq_switch(9d39400,0) at sleepq_switch+0xf7/frame 0x129e8c3c _sleep(9d39400,9d39440,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0x129e8c7c taskqueue_thread_loop(1e12bcc,129e8ce8) at taskqueue_thread_loop+0x9b/frame 0x129e8cb4 fork_exit(102ad60,1e12bcc,129e8ce8,0,0,...) at fork_exit+0x69/frame 0x129e8cd4 fork_trampoline() at 0xffc033de/frame 0x129e8cd4 --- kthread start Tracing command kernel pid 0 tid 100085 td 0x181cb180 sched_switch(181cb180,104) at sched_switch+0x4f3/frame 0xb78dbe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb78dc28 sleepq_switch(17ee5400,0) at sleepq_switch+0xf7/frame 0xb78dc48 msleep_spin_sbt(17ee5400,17ee5440,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb78dc80 taskqueue_thread_loop(1e1bea4,b78dce8) at taskqueue_thread_loop+0xd9/frame 0xb78dcb4 fork_exit(102ad60,1e1bea4,b78dce8,0,0,...) at fork_exit+0x69/frame 0xb78dcd4 fork_trampoline() at 0xffc033de/frame 0xb78dcd4 --- kthread start Tracing command kernel pid 0 tid 100091 td 0x18870700 sched_switch(18870700,104) at sched_switch+0x4f3/frame 0xb7b0be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7b0c28 sleepq_switch(18377c00,0) at sleepq_switch+0xf7/frame 0xb7b0c48 msleep_spin_sbt(18377c00,18377c40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7b0c80 taskqueue_thread_loop(1e220cc,b7b0ce8) at taskqueue_thread_loop+0xd9/frame 0xb7b0cb4 fork_exit(102ad60,1e220cc,b7b0ce8,0,0,...) at fork_exit+0x69/frame 0xb7b0cd4 fork_trampoline() at 0xffc033de/frame 0xb7b0cd4 --- kthread start Tracing command kernel pid 0 tid 100092 td 0x18870380 sched_switch(18870380,104) at sched_switch+0x4f3/frame 0xb7b5be8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7b5c28 sleepq_switch(18377c00,0) at sleepq_switch+0xf7/frame 0xb7b5c48 msleep_spin_sbt(18377c00,18377c40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7b5c80 taskqueue_thread_loop(1e220cc,b7b5ce8) at taskqueue_thread_loop+0xd9/frame 0xb7b5cb4 fork_exit(102ad60,1e220cc,b7b5ce8,0,0,...) at fork_exit+0x69/frame 0xb7b5cd4 fork_trampoline() at 0xffc033de/frame 0xb7b5cd4 --- kthread start Tracing command kernel pid 0 tid 100093 td 0x18870000 sched_switch(18870000,104) at sched_switch+0x4f3/frame 0xb7babe8 mi_switch(104) at mi_switch+0x1c4/frame 0xb7bac28 sleepq_switch(18377c00,0) at sleepq_switch+0xf7/frame 0xb7bac48 msleep_spin_sbt(18377c00,18377c40,1476273,0,0,0,0,100) at msleep_spin_sbt+0x132/frame 0xb7bac80 taskqueue_thread_loop(1e220cc,b7bace8) at taskqueue_thread_loop+0xd9/frame 0xb7bacb4 fork_exit(102ad60,1e220cc,b7bace8,0,0,...) at fork_exit+0x69/frame 0xb7bacd4 fork_trampoline() at 0xffc033de/frame 0xb7bacd4 --- kthread start Tracing command kernel pid 0 tid 100095 td 0x18863880 sched_switch(18863880,104) at sched_switch+0x4f3/frame 0xb7c4be0 mi_switch(104) at mi_switch+0x1c4/frame 0xb7c4c1c sleepq_switch(9d38900,0) at sleepq_switch+0xf7/frame 0xb7c4c3c _sleep(9d38900,9d38940,0,1476273,0,0,0,0,100) at _sleep+0x2f7/frame 0xb7c4c7c taskqueue_thread_loop(1b2f4ec,b7c4ce8) at taskqueue_thread_loop+0x9b/frame 0xb7c4cb4 fork_exit(102ad60,1b2f4ec,b7c4ce8,0,0,...) at fork_exit+0x69/frame 0xb7c4cd4 fork_trampoline() at 0xffc033de/frame 0xb7c4cd4 --- kthread start db:0:pho> dump Physical memory: 3428 MB Dumping 236 MB: 221 205 189 173 157 141 125 109 93 77 61 45 29 13 Dump complete db> reset (kgdb) bt #0 0x00fd4949 in doadump (textdump=0x0) at ../../../kern/kern_shutdown.c:392 #1 0x00a05154 in db_dump (dummy=0x1017c05, dummy2=0x0, dummy3=0xffffffff, dummy4=0x278a4584 "") at ../../../ddb/db_command.c:575 #2 0x00a04f1a in db_command (cmd_table=, dopager=) at ../../../ddb/db_command.c:482 #3 0x00a09956 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #4 0x00a04f1a in db_command (cmd_table=, dopager=) at ../../../ddb/db_command.c:482 #5 0x00a04c8c in db_command_loop () at ../../../ddb/db_command.c:535 #6 0x00a07d16 in db_trap (type=0x3, code=0x0) at ../../../ddb/db_main.c:252 #7 0x01018369 in kdb_trap (type=0x3, code=0x0, tf=0x278a487c) at ../../../kern/subr_kdb.c:689 #8 0x01417f9a in trap (frame=0x278a487c) at ../../../i386/i386/trap.c:668 #9 0xffc0316f in ?? () #10 0x01017c05 in kdb_enter (why=0x1523991 "panic", msg=0x1523991 "panic") at include/cpufunc.h:62 #11 0x00fd4af7 in vpanic (fmt=0x1463c56 "%s", ap=0x278a490c "a&O\001F\002 ") at ../../../kern/kern_shutdown.c:899 #12 0x00fd4914 in panic (fmt=0x1463c56 "%s") at ../../../kern/kern_shutdown.c:836 #13 0x014188a7 in trap_fatal (frame=, eva=) at ../../../i386/i386/trap.c:963 #14 0x01418922 in trap_pfault (frame=, usermode=, eva=0x187d6000, signo=0x0, ucode=0x0) at include/pcpu_aux.h:54 #15 0x01417fb8 in trap (frame=0x278a4a58) at ../../../i386/i386/trap.c:475 #16 0xffc0316f in ?? () #17 0x01046736 in sem_remove (semidx=0x0, cred=0x18d13e00) at ../../../kern/sysv_sem.c:588 #18 0x01045804 in kern_semctl (td=0x27300e00, semid=0x230000, semnum=0x0, cmd=0x0, arg=0x278a4bc0, rval=0x278a4bbc) at ../../../kern/sysv_sem.c:774 #19 0x010453b3 in sys___semctl (td=0x27300e00, uap=0x2730108c) at ../../../kern/sysv_sem.c:674 #20 0x01419016 in syscall (frame=0x278a4ce8) at subr_syscall.c:151 #21 0xffc033c9 in ?? () #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) f 17 #17 0x01046736 in sem_remove (semidx=0x0, cred=0x18d13e00) at ../../../kern/sysv_sem.c:588 588 sem[i] = sem[i + semakptr->u.sem_nsems]; (kgdb) l 583 if ((sema[i].u.sem_perm.mode & SEM_ALLOC) && 584 sema[i].u.__sem_base > semakptr->u.__sem_base) 585 mtx_lock_flags(&sema_mtx[i], LOP_DUPOK); 586 } 587 for (i = semakptr->u.__sem_base - sem; i < semtot; i++) 588 sem[i] = sem[i + semakptr->u.sem_nsems]; 589 for (i = 0; i < seminfo.semmni; i++) { 590 if ((sema[i].u.sem_perm.mode & SEM_ALLOC) && 591 sema[i].u.__sem_base > semakptr->u.__sem_base) { 592 sema[i].u.__sem_base -= semakptr->u.sem_nsems; (kgdb) info loc semakptr = (struct semid_kernel *) 0x187d4000 i = 0x9b (kgdb) p *semakptr $1 = {u = {sem_perm = {cuid = 0x0, cgid = 0xfffe, uid = 0x0, gid = 0xfffe, mode = 0x0, seq = 0x23, key = 0x3cd02d02}, __sem_base = 0x187d5000, sem_nsems = 0xba, sem_otime = 0x0, sem_ctime = 0x5e466027}, label = 0x0, cred = 0x0} (kgdb) p sema[0x9b] $2 = {u = {sem_perm = {cuid = 0xdeadc0de, cgid = 0xdeadc0de, uid = 0xdeadc0de, gid = 0xdeadc0de, mode = 0xc0de, seq = 0xdead, key = 0xdeadc0de}, __sem_base = 0xdeadc0de, sem_nsems = 0xc0de, sem_otime = 0xdeadc0de, sem_ctime = 0xdeadc0de}, label = 0xdeadc0de, cred = 0xdeadc0de} (kgdb) p seminfo $3 = {semmni = 0x32, semmns = 0x154, semmnu = 0x96, semmsl = 0x154, semopm = 0x64, semume = 0x32, semusz = 0x268, semvmx = 0x7fff, semaem = 0x4000} (kgdb) p semtot $4 = 0xba (kgdb) p sem $5 = (struct sem *) 0x187d5000 (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_tc.c =================================================================== --- /usr/src/sys/kern/kern_tc.c (revision 357822) +++ /usr/src/sys/kern/kern_tc.c (working copy) @@ -72,6 +72,7 @@ struct timehands { struct timecounter *th_counter; int64_t th_adjustment; uint64_t th_scale; + u_int th_large_delta; u_int th_offset_count; struct bintime th_offset; struct bintime th_bintime; @@ -87,6 +88,7 @@ static struct timehands ths[16] = { [0] = { .th_counter = &dummy_timecounter, .th_scale = (uint64_t)-1 / 1000000, + .th_large_delta = 1000000, .th_offset = { .sec = 1 }, .th_generation = 1, }, @@ -202,23 +204,75 @@ tc_delta(struct timehands *th) * the comment in for a description of these 12 functions. */ -#ifdef FFCLOCK -void -fbclock_binuptime(struct bintime *bt) +static __inline void +bintime_off(struct bintime *bt, u_int off) { struct timehands *th; - unsigned int gen; + struct bintime *btp; + uint64_t scale, x; + u_int delta, gen, large_delta; do { th = timehands; gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - bintime_addx(bt, th->th_scale * tc_delta(th)); + btp = (struct bintime *)((vm_offset_t)th + off); + *bt = *btp; + scale = th->th_scale; + delta = tc_delta(th); + large_delta = th->th_large_delta; atomic_thread_fence_acq(); } while (gen == 0 || gen != th->th_generation); + + if (__predict_false(delta >= large_delta)) { + /* Avoid overflow for scale * delta. */ + x = (scale >> 32) * delta; + bt->sec += x >> 32; + bintime_addx(bt, x << 32); + bintime_addx(bt, (scale & 0xffffffff) * delta); + } else { + bintime_addx(bt, scale * delta); + } } +#define GETTHBINTIME(dst, member) \ +do { \ + _Static_assert(_Generic(((struct timehands *)NULL)->member, \ + struct bintime: 1, default: 0) == 1, \ + "struct timehands member is not of struct bintime type"); \ + bintime_off(dst, __offsetof(struct timehands, member)); \ +} while (0) +static __inline void +getthmember(void *out, size_t out_size, u_int off) +{ + struct timehands *th; + u_int gen; + + do { + th = timehands; + gen = atomic_load_acq_int(&th->th_generation); + memcpy(out, (char *)th + off, out_size); + atomic_thread_fence_acq(); + } while (gen == 0 || gen != th->th_generation); +} +#define GETTHMEMBER(dst, member) \ +do { \ + _Static_assert(_Generic(*dst, \ + __typeof(((struct timehands *)NULL)->member): 1, \ + default: 0) == 1, \ + "*dst and struct timehands member have different types"); \ + getthmember(dst, sizeof(*dst), __offsetof(struct timehands, \ + member)); \ +} while (0) + +#ifdef FFCLOCK void +fbclock_binuptime(struct bintime *bt) +{ + + GETTHBINTIME(bt, th_offset); +} + +void fbclock_nanouptime(struct timespec *tsp) { struct bintime bt; @@ -239,16 +293,8 @@ fbclock_microuptime(struct timeval *tvp) void fbclock_bintime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_bintime); } void @@ -272,100 +318,55 @@ fbclock_microtime(struct timeval *tvp) void fbclock_getbinuptime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_offset); } void fbclock_getnanouptime(struct timespec *tsp) { - struct timehands *th; - unsigned int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timespec(&th->th_offset, tsp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timespec(&bt, tsp); } void fbclock_getmicrouptime(struct timeval *tvp) { - struct timehands *th; - unsigned int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timeval(&th->th_offset, tvp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timeval(&bt, tvp); } void fbclock_getbintime(struct bintime *bt) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_bintime); } void fbclock_getnanotime(struct timespec *tsp) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } void fbclock_getmicrotime(struct timeval *tvp) { - struct timehands *th; - unsigned int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tvp = th->th_microtime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tvp, th_microtime); } #else /* !FFCLOCK */ + void binuptime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_offset); } void @@ -389,16 +390,8 @@ microuptime(struct timeval *tvp) void bintime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - bintime_addx(bt, th->th_scale * tc_delta(th)); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHBINTIME(bt, th_bintime); } void @@ -422,85 +415,47 @@ microtime(struct timeval *tvp) void getbinuptime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_offset; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_offset); } void getnanouptime(struct timespec *tsp) { - struct timehands *th; - u_int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timespec(&th->th_offset, tsp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timespec(&bt, tsp); } void getmicrouptime(struct timeval *tvp) { - struct timehands *th; - u_int gen; + struct bintime bt; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - bintime2timeval(&th->th_offset, tvp); - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(&bt, th_offset); + bintime2timeval(&bt, tvp); } void getbintime(struct bintime *bt) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *bt = th->th_bintime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(bt, th_bintime); } void getnanotime(struct timespec *tsp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } void getmicrotime(struct timeval *tvp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tvp = th->th_microtime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tvp, th_microtime); } #endif /* FFCLOCK */ @@ -516,15 +471,8 @@ getboottime(struct timeval *boottime) void getboottimebin(struct bintime *boottimebin) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *boottimebin = th->th_boottime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(boottimebin, th_boottime); } #ifdef FFCLOCK @@ -1040,15 +988,8 @@ getmicrotime(struct timeval *tvp) void dtrace_getnanotime(struct timespec *tsp) { - struct timehands *th; - u_int gen; - do { - th = timehands; - gen = atomic_load_acq_int(&th->th_generation); - *tsp = th->th_nanotime; - atomic_thread_fence_acq(); - } while (gen == 0 || gen != th->th_generation); + GETTHMEMBER(tsp, th_nanotime); } /* @@ -1466,6 +1407,7 @@ tc_windup(struct bintime *new_boottimebin) scale += (th->th_adjustment / 1024) * 2199; scale /= th->th_counter->tc_frequency; th->th_scale = scale * 2; + th->th_large_delta = MIN(((uint64_t)1 << 63) / scale, UINT_MAX); /* * Now that the struct timehands is again consistent, set the new Index: /usr/src/sys/kern/sysv_sem.c =================================================================== --- /usr/src/sys/kern/sysv_sem.c (revision 357822) +++ /usr/src/sys/kern/sysv_sem.c (working copy) @@ -198,15 +198,15 @@ struct sem_undo { * semaphore info struct */ struct seminfo seminfo = { - SEMMNI, /* # of semaphore identifiers */ - SEMMNS, /* # of semaphores in system */ - SEMMNU, /* # of undo structures in system */ - SEMMSL, /* max # of semaphores per id */ - SEMOPM, /* max # of operations per semop call */ - SEMUME, /* max # of undo entries per process */ - SEMUSZ, /* size in bytes of undo structure */ - SEMVMX, /* semaphore maximum value */ - SEMAEM /* adjust on exit max value */ + .semmni = SEMMNI, /* # of semaphore identifiers */ + .semmns = SEMMNS, /* # of semaphores in system */ + .semmnu = SEMMNU, /* # of undo structures in system */ + .semmsl = SEMMSL, /* max # of semaphores per id */ + .semopm = SEMOPM, /* max # of operations per semop call */ + .semume = SEMUME, /* max # of undo entries per process */ + .semusz = SEMUSZ, /* size in bytes of undo structure */ + .semvmx = SEMVMX, /* semaphore maximum value */ + .semaem = SEMAEM, /* adjust on exit max value */ }; SYSCTL_INT(_kern_ipc, OID_AUTO, semmni, CTLFLAG_RDTUN, &seminfo.semmni, 0, @@ -558,8 +558,14 @@ sem_remove(int semidx, struct ucred *cred) int i; KASSERT(semidx >= 0 && semidx < seminfo.semmni, - ("semidx out of bounds")); + ("semidx out of bounds")); + mtx_assert(&sem_mtx, MA_OWNED); semakptr = &sema[semidx]; + KASSERT(semakptr->u.__sem_base - sem + semakptr->u.sem_nsems <= semtot, + ("sem_remove: sema %d corrupted sem pointer %p %p %d %d", + semidx, semakptr->u.__sem_base, sem, semakptr->u.sem_nsems, + semtot)); + semakptr->u.sem_perm.cuid = cred ? cred->cr_uid : 0; semakptr->u.sem_perm.uid = cred ? cred->cr_uid : 0; semakptr->u.sem_perm.mode = 0; -- Test scenario misc/syscall4.sh