GENERIC HEAD from 2011-12-08 03:20:48 UTC, r228351M, vmcore.6 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r228351M: Thu Dec 8 16:14:20 CET 2011 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3963445248 (3779 MB) : Trying to mount root from ufs:/dev/ada0s3a [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0s3a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s3a: clean, 685678 free (974 frags, 85588 blocks, 0.1% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 126186 free (4218 frags, 15246 blocks, 0.4% fragmentation) /dev/ada0s1g: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1g: clean, 114671044 free (1252 frags, 14333724 blocks, 0.0% fragmentation) /dev/ada0s3d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s3d: clean, 5868612 free (160412 frags, 713525 blocks, 1.6% fragmentation) /dev/ada0s3e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s3e: clean, 19782872 free (10680 frags, 2471524 blocks, 0.0% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0 fwe0 fwip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 Starting devd. Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: fwe0. fwe0: flags=8802 metric 0 mtu 1500 options=8 ether 02:dc:10:62:ad:eb nd6 options=29 ch 1 dma -1 Starting Network: fwip0. fwip0: flags=8802 metric 0 mtu 1500 lladdr 0.dc.10.0.1.62.ad.eb.a.2.ff.fe.0.0.0.0 nd6 options=29 add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 re0: link state changed to UP ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xffffff80ee9d1128 bufwait (bufwait) @ kern/vfs_bio.c:2658 2nd 0xfffffe000630e200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x854 _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x6a0 ufs_mkdir() at ufs_mkdir+0x4b4 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x284 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x8009408ec, rsp = 0x7fffffffd788, rbp = 0x800c07050 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Starting default moused. Starting inetd. Thu Dec 8 16:18:11 CET 2011 FreeBSD/amd64 (x4.osted.lan) (console) login: Dec 8 18:02:10 x4 su: pho to root on /dev/pts/0 Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0xffffff811518cdf0 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff80b2b91d stack pointer = 0x28:0xffffff81151e6690 frame pointer = 0x28:0xffffff81151e66b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 38230 (syscall4) [ thread pid 38230 tid 6069626 ] Stopped at cpu_thread_clean+0x2d: movq $0,0xe0(%rbx) db> run pho db:0:pho> bt Tracing pid 38230 tid 6069626 td 0xfffffe0006c5c000 cpu_thread_clean() at cpu_thread_clean+0x2d thread_free() at thread_free+0x2a create_thread() at create_thread+0x383 sys_thr_create() at sys_thr_create+0x79 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x800ad0d3c, rsp = 0x7ffffc9e4f68, rbp = 0x1ae --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x4c6580 curthread = 0xfffffe00028fd460: pid 11 "idle: cpu0" curpcb = 0xffffff8000232d10 fpcurthread = none idlethread = 0xfffffe00028fd460: tid 100003 "idle: cpu0" curpmap = 0xffffffff8114ad50 tssp = 0xffffffff8135f9e0 commontssp = 0xffffffff8135f9e0 rsp0 = 0xffffff8000232d10 gs32p = 0xffffffff8135db38 ldt = 0xffffffff8135db78 tss = 0xffffffff8135db68 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f107580 curthread = 0xfffffe00028fd000: pid 11 "idle: cpu1" curpcb = 0xffffff8000237d10 fpcurthread = none idlethread = 0xfffffe00028fd000: tid 100004 "idle: cpu1" curpmap = 0xffffffff8114ad50 tssp = 0xffffffff8135fa48 commontssp = 0xffffffff8135fa48 rsp0 = 0xffffff8000237d10 gs32p = 0xffffffff8135dba0 ldt = 0xffffffff8135dbe0 tss = 0xffffffff8135dbd0 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f10e580 curthread = 0xfffffe0006c5c000: pid 38230 "syscall4" curpcb = 0xffffff81151e6d10 fpcurthread = none idlethread = 0xfffffe00029088c0: tid 100005 "idle: cpu2" curpmap = 0xfffffe0006b0def8 tssp = 0xffffffff8135fab0 commontssp = 0xffffffff8135fab0 rsp0 = 0xffffff81151e6d10 gs32p = 0xffffffff8135dc08 ldt = 0xffffffff8135dc48 tss = 0xffffffff8135dc38 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f115580 curthread = 0xfffffe0002908460: pid 11 "idle: cpu3" curpcb = 0xffffff8000241d10 fpcurthread = none idlethread = 0xfffffe0002908460: tid 100006 "idle: cpu3" curpmap = 0xffffffff8114ad50 tssp = 0xffffffff8135fb18 commontssp = 0xffffffff8135fb18 rsp0 = 0xffffff8000241d10 gs32p = 0xffffffff8135dc70 ldt = 0xffffffff8135dcb0 tss = 0xffffffff8135dca0 spin locks held: db:0:allpcpu> show alllocks db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xfffffe00061f6ac8 /dev/ada0s3a on / (ufs) 0xfffffe00061f7000 devfs on /dev (devfs) 0xfffffe00061f6730 /dev/ada0s1f on /home (ufs) 0xfffffe00061f6398 /dev/ada0s1g on /tmp (ufs) 0xfffffe00061f6000 /dev/ada0s3d on /usr (ufs) 0xfffffe000629f730 /dev/ada0s3e on /var (ufs) 0xfffffe0006b2d398 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 38230 9172 2331 65534 T+ (threaded) syscall4 6069626 Run CPU 2 syscall4 6069618 s syscall4 9173 2426 2331 0 S+ nanslp 0xffffffff8114f128 sleep 9172 2426 2331 65534 S+ wait 0xfffffe0006986488 syscall4 2455 0 0 0 DL mdwait 0xfffffe000290e000 [md5] 2426 2331 2331 0 S+ wait 0xfffffe0006233910 sh 2331 2327 2331 0 S+ wait 0xfffffe00069fe910 sh 2327 2323 2327 0 S+ wait 0xfffffe00069fe488 bash 2323 2322 2323 0 S+ pause 0xfffffe00069fe0a0 csh 2322 2180 2322 1001 S+ wait 0xfffffe0006988910 su 2180 2179 2180 1001 Ss+ wait 0xfffffe000698b910 bash 2179 2177 2177 1001 S select 0xfffffe0006649a40 sshd 2177 2007 2177 0 Ss select 0xfffffe0006649d40 sshd 2130 1 2130 0 Ss+ ttyin 0xfffffe0002def8a8 getty 2129 1 2129 0 Ss+ ttyin 0xfffffe0002defca8 getty 2128 1 2128 0 Ss+ ttyin 0xfffffe0002df00a8 getty 2127 1 2127 0 Ss+ ttyin 0xfffffe0002df04a8 getty 2126 1 2126 0 Ss+ ttyin 0xfffffe000293d8a8 getty 2125 1 2125 0 Ss+ ttyin 0xfffffe0002b178a8 getty 2124 1 2124 0 Ss+ ttyin 0xfffffe000293d4a8 getty 2123 1 2123 0 Ss+ ttyin 0xfffffe000293c4a8 getty 2122 1 2122 0 Ss+ ttyin 0xfffffe0002b170a8 getty 2083 1 2083 0 Ss select 0xfffffe00066e8240 inetd 2050 1 2050 0 Ss select 0xfffffe00028ed3c0 moused 2026 1 2026 0 Ss nanslp 0xffffffff8114f128 cron 2019 1 2019 25 Ss pause 0xfffffe00063699b0 sendmail 2015 1 2015 0 Ss select 0xfffffe0006649440 sendmail 2007 1 2007 0 Ss select 0xfffffe00028ed4c0 sshd 1961 1 1961 0 Ss select 0xfffffe00066494c0 ntpd 1832 1831 1831 0 S (threaded) nfsd 100134 S rpcsvc 0xfffffe00061f1ca0 nfsd: service 100133 S rpcsvc 0xfffffe00061f1d20 nfsd: service 100132 S rpcsvc 0xfffffe00061f1da0 nfsd: service 100093 S rpcsvc 0xfffffe0006334da0 nfsd: master 1831 1 1831 0 Ss select 0xfffffe00028ed640 nfsd 1822 1 1822 0 Ss select 0xfffffe00028ec340 mountd 1724 1 1724 0 Ss select 0xfffffe00028edac0 rpcbind 1698 1 1698 0 Ss select 0xfffffe00028ec4c0 syslogd 1490 1 1490 0 Ss select 0xfffffe00028ec6c0 devd 18 0 0 0 DL sdflush 0xffffffff81343a78 [softdepflush] 17 0 0 0 DL syncer 0xffffffff8133ad20 [syncer] 16 0 0 0 DL vlruwt 0xfffffe0006162000 [vnlru] 9 0 0 0 DL psleep 0xffffffff8133a7a8 [bufdaemon] 8 0 0 0 DL pgzero 0xffffffff8134d8bc [pagezero] 7 0 0 0 DL psleep 0xffffffff8134cb68 [vmdaemon] 6 0 0 0 DL psleep 0xffffffff8134cb2c [pagedaemon] 5 0 0 0 DL ccb_scan 0xffffffff8110e960 [xpt_thrd] 4 0 0 0 DL waiting_ 0xffffffff8133dd80 [sctp_iterator] 3 0 0 0 DL - 0xfffffe0002da0048 [fdc0] 2 0 0 0 SL - 0xffffff8000866000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xffffff8000863e18 [usbus5] 100057 D - 0xffffff8000863dc0 [usbus5] 100056 D - 0xffffff8000863d68 [usbus5] 100055 D - 0xffffff8000863d10 [usbus5] 100053 D - 0xffffff800085b460 [usbus4] 100052 D - 0xffffff800085b408 [usbus4] 100051 D - 0xffffff800085b3b0 [usbus4] 100050 D - 0xffffff800085b358 [usbus4] 100049 D - 0xffffff8000858460 [usbus3] 100048 D - 0xffffff8000858408 [usbus3] 100047 D - 0xffffff80008583b0 [usbus3] 100046 D - 0xffffff8000858358 [usbus3] 100045 D - 0xffffff8000855460 [usbus2] 100044 D - 0xffffff8000855408 [usbus2] 100043 D - 0xffffff80008553b0 [usbus2] 100042 D - 0xffffff8000855358 [usbus2] 100040 D - 0xffffff8000852460 [usbus1] 100039 D - 0xffffff8000852408 [usbus1] 100038 D - 0xffffff80008523b0 [usbus1] 100037 D - 0xffffff8000852358 [usbus1] 100035 D - 0xffffff800084f460 [usbus0] 100034 D - 0xffffff800084f408 [usbus0] 100033 D - 0xffffff800084f3b0 [usbus0] 100032 D - 0xffffff800084f358 [usbus0] 14 0 0 0 DL - 0xffffffff8114ddc4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100015 D - 0xffffffff8114a230 [g_down] 100014 D - 0xffffffff8114a228 [g_up] 100013 D - 0xffffffff8114a218 [g_event] 12 0 0 0 RL (threaded) [intr] 100066 I [irq12: psm0] 100065 I [irq1: atkbd0] 100063 I [swi0: uart] 100060 I [irq20: fwohci0] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100026 I [swi5: +] 100020 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi3: vm] 100011 I [swi4: clock] 100010 RunQ [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe00028fb910 [init] 10 0 0 0 DL audit_wo 0xffffffff813429f0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100077 D - 0xfffffe000611f100 [mca taskq] 100067 D - 0xffffffff8114ddc4 [deadlkres] 100061 D - 0xfffffe0002d8db80 [fw0_taskq] 100027 D - 0xfffffe0002a68b00 [thread taskq] 100025 D - 0xfffffe0002a68c00 [acpi_task_2] 100024 D - 0xfffffe0002a68c00 [acpi_task_1] 100023 D - 0xfffffe0002a68c00 [acpi_task_0] 100022 D - 0xfffffe0002a68c80 [kqueue taskq] 100021 D - 0xfffffe0002a68d00 [ffs_trim taskq] 100016 D - 0xfffffe00028cea00 [firmware taskq] 100000 D sched 0xffffffff8114a360 [swapper] db:0:ps> allt Tracing command syscall4 pid 38230 tid 6069626 td 0xfffffe0006c5c000 cpu_thread_clean() at cpu_thread_clean+0x2d thread_free() at thread_free+0x2a create_thread() at create_thread+0x383 sys_thr_create() at sys_thr_create+0x79 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x800ad0d3c, rsp = 0x7ffffc9e4f68, rbp = 0x1ae --- Tracing command syscall4 pid 38230 tid 6069618 td 0xfffffe0006a08460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea thread_suspend_switch() at thread_suspend_switch+0x103 thread_single() at thread_single+0x222 exit1() at exit1+0x81 sys_sys_exit() at sys_sys_exit+0xe amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x800b13ccc, rsp = 0x7ffffcfe76c8, rbp = 0x80100d000 --- Tracing command sleep pid 9173 tid 5212414 td 0xfffffe0006c9a000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24d kern_nanosleep() at kern_nanosleep+0x118 sys_nanosleep() at sys_nanosleep+0x6e amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80092f67c, rsp = 0x7fffffffd688, rbp = 0x7fffffffd700 --- Tracing command syscall4 pid 9172 tid 5328478 td 0xfffffe0006cb88c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ae699c, rsp = 0x7fffffffd518, rbp = 0x7fffffffd740 --- Tracing command md5 pid 2455 tid 100175 td 0xfffffe0006843000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 md_kthread() at md_kthread+0x16a fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811511ed00, rbp = 0 --- Tracing command sh pid 2426 tid 100081 td 0xfffffe000623a8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d5799c, rsp = 0x7fffffffd458, rbp = 0x7fffffffd628 --- Tracing command sh pid 2331 tid 100176 td 0xfffffe000660f8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d5799c, rsp = 0x7fffffffd398, rbp = 0x7fffffffd568 --- Tracing command bash pid 2327 tid 100177 td 0xfffffe000660f460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800cab99c, rsp = 0x7fffffffd648, rbp = 0x8010ce220 --- Tracing command csh pid 2323 tid 100178 td 0xfffffe00069e6000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_sigsuspend() at kern_sigsuspend+0x9b sys_sigsuspend() at sys_sigsuspend+0x34 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800d6dadc, rsp = 0x7fffffffd328, rbp = 0x801413600 --- Tracing command su pid 2322 tid 100165 td 0xfffffe00069c3000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ef999c, rsp = 0x7fffffffd468, rbp = 0x913 --- Tracing command bash pid 2180 tid 100162 td 0xfffffe0006364000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800cab99c, rsp = 0x7fffffffd8c8, rbp = 0x8010e04a0 --- Tracing command sshd pid 2179 tid 100142 td 0xfffffe000660e000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8025079fc, rsp = 0x7fffffffcc68, rbp = 0x7fffffffccf0 --- Tracing command sshd pid 2177 tid 100082 td 0xfffffe000623a460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac sys_poll() at sys_poll+0x2b6 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024a432c, rsp = 0x7fffffffcd08, rbp = 0x8030171c0 --- Tracing command getty pid 2130 tid 100152 td 0xfffffe00066448c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2129 tid 100154 td 0xfffffe0006644000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2128 tid 100155 td 0xfffffe00068b8460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2127 tid 100153 td 0xfffffe0006644460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2126 tid 100156 td 0xfffffe00068b8000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2125 tid 100157 td 0xfffffe000686f8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2124 tid 100159 td 0xfffffe000686f000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2123 tid 100160 td 0xfffffe000686e8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command getty pid 2122 tid 100158 td 0xfffffe000686f460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x6c sys_read() at sys_read+0x55 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b64a7c, rsp = 0x7fffffffdca8, rbp = 0x3 --- Tracing command inetd pid 2083 tid 100116 td 0xfffffe0006390460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800f7a9fc, rsp = 0x7fffffffcd78, rbp = 0x1 --- Tracing command moused pid 2050 tid 100137 td 0xfffffe0006158460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d8a9fc, rsp = 0x7fffffffd858, rbp = 0x7fffffffdf31 --- Tracing command cron pid 2026 tid 100112 td 0xfffffe00063918c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24d kern_nanosleep() at kern_nanosleep+0x118 sys_nanosleep() at sys_nanosleep+0x6e amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d5167c, rsp = 0x7fffffffdae8, rbp = 0x3c --- Tracing command sendmail pid 2019 tid 100103 td 0xfffffe00062d48c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_sigsuspend() at kern_sigsuspend+0x9b sys_sigsuspend() at sys_sigsuspend+0x34 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x801360adc, rsp = 0x7fffffffbdb8, rbp = 0x1 --- Tracing command sendmail pid 2015 tid 100143 td 0xfffffe00066518c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8014039fc, rsp = 0x7fffffffb228, rbp = 0x7fffffffb2c0 --- Tracing command sshd pid 2007 tid 100144 td 0xfffffe0006651460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8025079fc, rsp = 0x7fffffffcd88, rbp = 0x2 --- Tracing command ntpd pid 1961 tid 100147 td 0xfffffe0006650460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80137b9fc, rsp = 0x7fffffffdbd8, rbp = 0x7fffffffdd08 --- Tracing command nfsd pid 1832 tid 100134 td 0xfffffe00063b0000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8008b071c, rsp = 0x7fffffffd698, rbp = 0x5 --- Tracing command nfsd pid 1832 tid 100133 td 0xfffffe00063b0460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8008b071c, rsp = 0x7fffffffd698, rbp = 0x5 --- Tracing command nfsd pid 1832 tid 100132 td 0xfffffe00063b08c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x8008b071c, rsp = 0x7fffffffd698, rbp = 0x5 --- Tracing command nfsd pid 1832 tid 100093 td 0xfffffe0006236000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_run() at svc_run+0x8f nfsrvd_nfsd() at nfsrvd_nfsd+0x1c7 nfssvc_nfsd() at nfssvc_nfsd+0x9b sys_nfssvc() at sys_nfssvc+0xab amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x8008b071c, rsp = 0x7fffffffd698, rbp = 0x5 --- Tracing command nfsd pid 1831 tid 100111 td 0xfffffe0006392000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8009529fc, rsp = 0x7fffffffd938, rbp = 0x7fffffffdc10 --- Tracing command mountd pid 1822 tid 100113 td 0xfffffe0006391460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b699fc, rsp = 0x7fffffffdb88, rbp = 0x801086060 --- Tracing command rpcbind pid 1724 tid 100114 td 0xfffffe0006391000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 sys_poll() at sys_poll+0x2b6 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800d0f32c, rsp = 0x7fffffffba58, rbp = 0x8010640c0 --- Tracing command syslogd pid 1698 tid 100107 td 0xfffffe000639a460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b679fc, rsp = 0x7fffffffd128, rbp = 0x80108c0d0 --- Tracing command devd pid 1490 tid 100098 td 0xfffffe0006393000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x6f0 sys_select() at sys_select+0x5d amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x449a3c, rsp = 0x7fffffffd838, rbp = 0x7fffffffd850 --- Tracing command softdepflush pid 18 tid 100076 td 0xfffffe000615b000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 softdep_flush() at softdep_flush+0x264 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c2cd00, rbp = 0 --- Tracing command syncer pid 17 tid 100075 td 0xfffffe000615b460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x541 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c27d00, rbp = 0 --- Tracing command vnlru pid 16 tid 100074 td 0xfffffe000615b8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 vnlru_proc() at vnlru_proc+0x586 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c22d00, rbp = 0 --- Tracing command bufdaemon pid 9 tid 100073 td 0xfffffe000615d000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 buf_daemon() at buf_daemon+0x192 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c1dd00, rbp = 0 --- Tracing command pagezero pid 8 tid 100072 td 0xfffffe0002dbc000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c18d00, rbp = 0 --- Tracing command vmdaemon pid 7 tid 100071 td 0xfffffe0002dbc460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c13d00, rbp = 0 --- Tracing command pagedaemon pid 6 tid 100070 td 0xfffffe0002dbc8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 vm_pageout() at vm_pageout+0x8cb fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c0ed00, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100069 td 0xfffffe0002dbe000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112ba9d00, rbp = 0 --- Tracing command sctp_iterator pid 4 tid 100068 td 0xfffffe0002dbe460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 sctp_iterator_thread() at sctp_iterator_thread+0x3f fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112ba4d00, rbp = 0 --- Tracing command fdc0 pid 3 tid 100064 td 0xfffffe0002dc08c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 fdc_thread() at fdc_thread+0x74a fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112b8cd00, rbp = 0 --- Tracing command fw0_probe pid 2 tid 100062 td 0xfffffe0002dc1460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a fw_bus_probe_thread() at fw_bus_probe_thread+0xbc fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112b74d00, rbp = 0 --- Tracing command usb pid 15 tid 100058 td 0xfffffe0002d86000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112813d00, rbp = 0 --- Tracing command usb pid 15 tid 100057 td 0xfffffe0002d86460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811280ed00, rbp = 0 --- Tracing command usb pid 15 tid 100056 td 0xfffffe0002d868c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112809d00, rbp = 0 --- Tracing command usb pid 15 tid 100055 td 0xfffffe0002d87000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112804d00, rbp = 0 --- Tracing command usb pid 15 tid 100053 td 0xfffffe0002d878c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112677d00, rbp = 0 --- Tracing command usb pid 15 tid 100052 td 0xfffffe0002d88000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112672d00, rbp = 0 --- Tracing command usb pid 15 tid 100051 td 0xfffffe0002d88460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811266dd00, rbp = 0 --- Tracing command usb pid 15 tid 100050 td 0xfffffe0002d888c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112668d00, rbp = 0 --- Tracing command usb pid 15 tid 100049 td 0xfffffe0002d76000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811261fd00, rbp = 0 --- Tracing command usb pid 15 tid 100048 td 0xfffffe0002d76460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811261ad00, rbp = 0 --- Tracing command usb pid 15 tid 100047 td 0xfffffe0002d768c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112615d00, rbp = 0 --- Tracing command usb pid 15 tid 100046 td 0xfffffe0002d77000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112610d00, rbp = 0 --- Tracing command usb pid 15 tid 100045 td 0xfffffe0002d77460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81125c7d00, rbp = 0 --- Tracing command usb pid 15 tid 100044 td 0xfffffe0002d778c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81125c2d00, rbp = 0 --- Tracing command usb pid 15 tid 100043 td 0xfffffe0002d79000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81125bdd00, rbp = 0 --- Tracing command usb pid 15 tid 100042 td 0xfffffe0002d79460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81125b8d00, rbp = 0 --- Tracing command usb pid 15 tid 100040 td 0xfffffe0002d7a000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811256ad00, rbp = 0 --- Tracing command usb pid 15 tid 100039 td 0xfffffe0002d6f000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112565d00, rbp = 0 --- Tracing command usb pid 15 tid 100038 td 0xfffffe0002d6f460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112560d00, rbp = 0 --- Tracing command usb pid 15 tid 100037 td 0xfffffe0002d6f8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811255bd00, rbp = 0 --- Tracing command usb pid 15 tid 100035 td 0xfffffe0002d70460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811250dd00, rbp = 0 --- Tracing command usb pid 15 tid 100034 td 0xfffffe0002d708c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112508d00, rbp = 0 --- Tracing command usb pid 15 tid 100033 td 0xfffffe0002d72000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112503d00, rbp = 0 --- Tracing command usb pid 15 tid 100032 td 0xfffffe0002d72460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x16b fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81124fed00, rbp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xfffffe000292f8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000278d00, rbp = 0 --- Tracing command geom pid 13 tid 100015 td 0xfffffe0002915000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 g_io_schedule_down() at g_io_schedule_down+0x228 g_down_procbody() at g_down_procbody+0x72 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800026ed00, rbp = 0 --- Tracing command geom pid 13 tid 100014 td 0xfffffe0002915460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 g_io_schedule_up() at g_io_schedule_up+0x10a g_up_procbody() at g_up_procbody+0x72 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000269d00, rbp = 0 --- Tracing command geom pid 13 tid 100013 td 0xfffffe00029158c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 g_run_events() at g_run_events+0x4a8 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000264d00, rbp = 0 --- Tracing command intr pid 12 tid 100066 td 0xfffffe0002dc0000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100065 td 0xfffffe0002dc0460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112b91d00, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xfffffe0002dc1000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112b7dd00, rbp = 0 --- Tracing command intr pid 12 tid 100060 td 0xfffffe0002d7a460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112840d00, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffffe0002d7a8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff811283bd00, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffffe0002d87460 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffffe0002d798c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81125b3d00, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xfffffe0002d70000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112556d00, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xfffffe0002d728c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81124f9d00, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xfffffe0002931460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80003bdd00, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xfffffe00029318c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80003b8d00, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffffe0002ae7000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80002afd00, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xfffffe0002ae78c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xfffffe000292e8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000287d00, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffffe000292f000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000282d00, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xfffffe000292f460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800027dd00, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xfffffe0002917000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xfffffe0002917460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800025ad00, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xfffffe00029178c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000255d00, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xfffffe00028fe460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000250d00, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xfffffe00028fe8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800024bd00, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xfffffe0002908000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea ithread_loop() at ithread_loop+0x25d fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000246d00, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe0002908460 cpustop_handler() at cpustop_handler+0x2b ipi_nmi_handler() at ipi_nmi_handler+0x50 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff8031bd0b, rsp = 0xffffff8000221fe0, rbp = 0xffffff8000241940 --- acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x1b binuptime() at binuptime+0x3b timercb() at timercb+0x1f5 lapic_handle_timer() at lapic_handle_timer+0xca Xtimerint() at Xtimerint+0x88 --- interrupt, rip = 0xffffffff80b0feb6, rsp = 0xffffff8000241ae0, rbp = 0xffffff8000241af0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x24e cpu_idle_acpi() at cpu_idle_acpi+0x45 cpu_idle() at cpu_idle+0x6a sched_idletd() at sched_idletd+0x27e fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00029088c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sched_idletd() at sched_idletd+0x2b0 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800023cd00, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00028fd000 cpustop_handler() at cpustop_handler+0x2b ipi_nmi_handler() at ipi_nmi_handler+0x50 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80b0feb6, rsp = 0xffffff8000213fe0, rbp = 0xffffff8000237af0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x24e cpu_idle_acpi() at cpu_idle_acpi+0x45 cpu_idle() at cpu_idle+0x6a sched_idletd() at sched_idletd+0x27e fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000237d00, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00028fd460 cpustop_handler() at cpustop_handler+0x2b ipi_nmi_handler() at ipi_nmi_handler+0x50 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff8031bd16, rsp = 0xffffffff8136d4c0, rbp = 0xffffff8000232ad0 --- acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x26 binuptime() at binuptime+0x3b cpu_activeclock() at cpu_activeclock+0x9d cpu_idle() at cpu_idle+0x7a sched_idletd() at sched_idletd+0x27e fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000232d00, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00028fd8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a2 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3eb sys_wait4() at sys_wait4+0x35 amd64_syscall() at amd64_syscall+0x27b Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x412f9c, rsp = 0x7fffffffd7a8, rbp = 0x7d --- Tracing command audit pid 10 tid 100001 td 0xfffffe00028fe000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000228d00, rbp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xfffffe000615a8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112c31d00, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffffe0002dbe8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 deadlkres() at deadlkres+0x2ac fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112b9fd00, rbp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xfffffe0002dc18c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5f fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8112aafd00, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe0002ae7460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80002aad00, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe0002aea000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5f fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80002a0d00, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe0002aea460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5f fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800029bd00, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffffe0002aea8c0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5f fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000296d00, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffffe000292e000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000291d00, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffffe000292e460 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800028cd00, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe0002931000 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000273d00, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff8114a7f0 sched_switch() at sched_switch+0x157 mi_switch() at mi_switch+0x1ea sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 scheduler() at scheduler+0x2a1 mi_startup() at mi_startup+0x77 btext() at btext+0x2c db:0:allt> call doadump Dumping 275 out of 3947 MB:..6%..12%..24%..35%..41%..53%..64%..76%..82%..94% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) bt #0 doadump (textdump=0x28fc000) at ../../../kern/kern_shutdown.c:261 #1 0xffffffff802f530c in db_fncall (dummy1=Variable "dummy1" is not available. ) at ../../../ddb/db_command.c:572 #2 0xffffffff802f55bd in db_command (last_cmdp=0xffffffff811100e0, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:448 #3 0xffffffff802fa033 in db_script_exec (scriptname=0xffffffff81110980 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff802f5641 in db_command (last_cmdp=0xffffffff811100e0, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:448 #5 0xffffffff802f5890 in db_command_loop () at ../../../ddb/db_command.c:501 #6 0xffffffff802f79e9 in db_trap (type=Variable "type" is not available. ) at ../../../ddb/db_main.c:229 #7 0xffffffff80871911 in kdb_trap (type=0xc, code=0x0, tf=0xffffff81151e65e0) at ../../../kern/subr_kdb.c:625 #8 0xffffffff80b2a71d in trap_fatal (frame=0xffffff81151e65e0, eva=Variable "eva" is not available. ) at ../../../amd64/amd64/trap.c:814 #9 0xffffffff80b2aa3e in trap_pfault (frame=0xffffff81151e65e0, usermode=0x0) at ../../../amd64/amd64/trap.c:735 #10 0xffffffff80b2ae4b in trap (frame=0xffffff81151e65e0) at ../../../amd64/amd64/trap.c:474 #11 0xffffffff80b14f3f in calltrap () at ../../../amd64/amd64/exception.S:228 #12 0xffffffff80b2b91d in cpu_thread_clean (td=Variable "td" is not available. ) at ../../../amd64/amd64/vm_machdep.c:294 #13 0xffffffff8084ad7a in thread_free (td=0xfffffe0006e4c8c0) at ../../../kern/kern_thread.c:366 #14 0xffffffff80849863 in create_thread (td=0xfffffe0006c5c000, ctx=0xffffff81151e67a0, start_func=0, arg=0x0, stack_base=0x0, stack_size=0x0, tls_base=0x0, child_tid=0x72, parent_tid=0x0, flags=0x693f, rtp=0x0) at ../../../kern/kern_thr.c:217 #15 0xffffffff80849a59 in sys_thr_create (td=0xfffffe0006c5c000, uap=0xffffff81151e6bc0) at ../../../kern/kern_thr.c:110 #16 0xffffffff80b29e5b in amd64_syscall (td=0xfffffe0006c5c000, traced=0x0) at subr_syscall.c:131 #17 0xffffffff80b15227 in Xfast_syscall () at ../../../amd64/amd64/exception.S:387 #18 0x0000000800ad0d3c in ?? () (kgdb) f 14 #14 0xffffffff80849863 in create_thread (td=0xfffffe0006c5c000, ctx=0xffffff81151e67a0, start_func=0, arg=0x0, stack_base=0x0, stack_size=0x0, tls_base=0x0, child_tid=0x72, parent_tid=0x0, flags=0x693f, rtp=0x0) at ../../../kern/kern_thr.c:217 217 thread_free(newtd); (kgdb) l 212 */ 213 if ((child_tid != NULL && 214 suword_lwpid(child_tid, newtd->td_tid)) || 215 (parent_tid != NULL && 216 suword_lwpid(parent_tid, newtd->td_tid))) { 217 thread_free(newtd); 218 error = EFAULT; 219 goto fail; 220 } 221 (kgdb) f 13 #13 0xffffffff8084ad7a in thread_free (td=0xfffffe0006e4c8c0) at ../../../kern/kern_thread.c:366 366 cpu_thread_free(td); (kgdb) l 361 362 lock_profile_thread_exit(td); 363 if (td->td_cpuset) 364 cpuset_rel(td->td_cpuset); 365 td->td_cpuset = NULL; 366 cpu_thread_free(td); 367 if (td->td_kstack != 0) 368 vm_thread_dispose(td); 369 uma_zfree(thread_zone, td); 370 } (kgdb) f 12 #12 0xffffffff80b2b91d in cpu_thread_clean (td=Variable "td" is not available. ) at ../../../amd64/amd64/vm_machdep.c:294 294 kmem_free(kernel_map, (vm_offset_t)pcb->pcb_tssp, (kgdb) l 289 290 /* 291 * Clean TSS/iomap 292 */ 293 if (pcb->pcb_tssp != NULL) { 294 kmem_free(kernel_map, (vm_offset_t)pcb->pcb_tssp, 295 ctob(IOPAGES + 1)); 296 pcb->pcb_tssp = NULL; 297 } 298 } (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/kern/kern_umtx.c =================================================================== --- /usr/src/sys/kern/kern_umtx.c (revision 228351) +++ /usr/src/sys/kern/kern_umtx.c (working copy) @@ -2441,7 +2441,7 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw uint32_t flags, wrflags; int32_t state, oldstate; int32_t blocked_readers; - int error; + int error, loops; uq = td->td_umtxq; flags = fuword32(&rwlock->rw_flags); @@ -2453,6 +2453,7 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw if (!(fflag & URWLOCK_PREFER_READER) && !(flags & URWLOCK_PREFER_READER)) wrflags |= URWLOCK_WRITE_WAITERS; + loops = 0; for (;;) { state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state)); /* try to lock it */ @@ -2496,6 +2497,10 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); + if (loops++) { + umtx_key_release(&uq->uq_key); + return (EINVAL); + } continue; } @@ -2577,7 +2582,7 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw int32_t state, oldstate; int32_t blocked_writers; int32_t blocked_readers; - int error; + int error, loops; uq = td->td_umtxq; flags = fuword32(&rwlock->rw_flags); @@ -2586,6 +2591,7 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw return (error); blocked_readers = 0; + loops = 0; for (;;) { state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state)); while (!(state & URWLOCK_WRITE_OWNER) && URWLOCK_READER_COUNT(state) == 0) { @@ -2633,6 +2639,10 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); + if (loops++) { + umtx_key_release(&uq->uq_key); + return (EINVAL); + } continue; } sleep: -- syscall4.sh