GENERIC HEAD from 2009-08-15 18:37:06 UTC, r196259M, vmcore.11 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-BETA2 #1 r196259M: Sun Aug 16 16:17:11 CEST 2009 pho@x4.osted.lan:/usr/src/sys/amd64/compile/PHO WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3974787072 (3790 MB) : Trying to mount root from ufs:/dev/ad4s2a Entropy harvesting: interrupts ethernet point_to_point kickstart. /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 637992 free (888 frags, 79638 blocks, 0.1% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 50599237 free (325 frags, 6324864 blocks, 0.0% fragmentation) /dev/ad4s2f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2f: clean, 716498 free (137546 frags, 72369 blocks, 2.8% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 45186370 free (46882 frags, 5642436 blocks, 0.1% fragmentation) Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. add net default: gateway 192.168.1.1 Configuring syscons: keymap blanktime. Local package initialization:lock order reversal: 1st 0xffffff0004428578 ufs (ufs) @ kern/vfs_mount.c:1200 2nd 0xffffff00044aa578 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1031 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd03 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 softdep_flushworklist() at softdep_flushworklist+0x61 ffs_sync() at ffs_sync+0x43a dounmount() at dounmount+0x2ca unmount() at unmount+0x27e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a09ec, rsp = 0x7fffffffe558, rbp = 0 --- ** /dev/ad4s2e ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes lock order reversal: 1st 0xffffff804fc18178 bufwait (bufwait) @ kern/vfs_bio.c:2559 2nd 0xffffff0004535200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x5e4 ufs_makeinode() at ufs_makeinode+0x2d5 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800c74dfc, rsp = 0x7fffffffe258, rbp = 0xa --- Aug 16 16:22:07 x4 su: pho to root on /dev/pts/0 ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 5479 files, 177838 used, 50599237 free (325 frags, 6324864 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** usage: kill [-s signal_name] pid ... kill -l [exit_status] kill -signal_name pid ... kill -signal_number pid ... fsck -y /tmp. Sun Aug 16 16:22:52 CEST 2009 FreeBSD/amd64 (x4.osted.lan) (console) login: panic: pmap_release: pmap resident count -1 != 0 cpuid = 1 KDB: enter: panic [thread pid 1519 tid 100110 ] Stopped at kdb_enter+0x3d: movq $0,0x68f9e0(%rip) db> run pho db:0:pho> bt Tracing pid 1519 tid 100110 td 0xffffff0004465000 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b pmap_release() at pmap_release+0xb0 vmspace_exit() at vmspace_exit+0xcf exit1() at exit1+0x376 sys_exit() at sys_exit+0xe syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x42397c, rsp = 0x7fffffffe7b8, rbp = 0x7fffffffe7f0 --- db:0:bt> show allpcpu Current CPU: 1 cpuid = 0 dynamic pcpu = 0x4d8480 curthread = 0xffffff00024ef000: pid 12 "swi4: clock" curpcb = 0xffffff800004ed40 fpcurthread = none idlethread = 0xffffff00024ef720: pid 11 "idle: cpu0" curpmap = 0 tssp = 0xffffffff80e2e380 commontssp = 0xffffffff80e2e380 rsp0 = 0xffffff800004ed40 gs32p = 0xffffffff80e2d1b8 ldt = 0xffffffff80e2d1f8 tss = 0xffffffff80e2d1e8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f43c480 curthread = 0xffffff0004465000: pid 1519 "gcc" curpcb = 0xffffff8075a70d40 fpcurthread = none idlethread = 0xffffff00024e3000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff80e2e3e8 commontssp = 0xffffffff80e2e3e8 rsp0 = 0xffffff8075a70d40 gs32p = 0xffffffff80e2d220 ldt = 0xffffffff80e2d260 tss = 0xffffffff80e2d250 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f443480 curthread = 0xffffff00024e4ab0: pid 12 "swi4: clock" curpcb = 0xffffff8000053d40 fpcurthread = none idlethread = 0xffffff00024e3390: pid 11 "idle: cpu2" curpmap = 0 tssp = 0xffffffff80e2e450 commontssp = 0xffffffff80e2e450 rsp0 = 0xffffff8000053d40 gs32p = 0xffffffff80e2d288 ldt = 0xffffffff80e2d2c8 tss = 0xffffffff80e2d2b8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f44a480 curthread = 0xffffff00024e3720: pid 11 "idle: cpu3" curpcb = 0xffffff8000035d40 fpcurthread = none idlethread = 0xffffff00024e3720: pid 11 "idle: cpu3" curpmap = 0 tssp = 0xffffffff80e2e4b8 commontssp = 0xffffffff80e2e4b8 rsp0 = 0xffffff8000035d40 gs32p = 0xffffffff80e2d2f0 ldt = 0xffffffff80e2d330 tss = 0xffffffff80e2d320 spin locks held: db:0:allpcpu> show alllocks Process 1141 (sshd) thread 0xffffff0004464ab0 (100111) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00046e1e40) locked @ kern/uipc_sockbuf.c:148 Process 1140 (sshd) thread 0xffffff00043d8ab0 (100084) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004940e40) locked @ kern/uipc_sockbuf.c:148 Process 1139 (sshd) thread 0xffffff0004888000 (100142) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004551b98) locked @ kern/uipc_sockbuf.c:148 Process 1032 (sshd) thread 0xffffff00044a4390 (100092) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004551648) locked @ kern/uipc_sockbuf.c:148 db:0:alllocks> show lockedvnods Locked vnodes db:0:lockedvnods> show mount 0xffffff0004383710 /dev/ad4s2a on / (ufs) 0xffffff0004372000 devfs on /dev (devfs) 0xffffff0004371a98 /dev/ad4s2f on /usr (ufs) 0xffffff0004371710 /dev/ad4s2d on /var (ufs) 0xffffff0004371388 /dev/ad4s2e on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 1519 1514 1134 1001 RE+ CPU 1 gcc 1518 1151 1147 1001 S nanslp 0xffffffff80c23668 sleep 1514 1492 1134 1001 S+ wait 0xffffff00047dc8c0 perl5.8.9 1492 1465 1134 1001 S+ wait 0xffffff009d0f48c0 make 1465 1246 1134 1001 S+ wait 0xffffff00047f6460 sh 1246 1243 1134 1001 S+ wait 0xffffff00045328c0 make 1243 1134 1134 1001 S+ wait 0xffffff00046b1460 perl5.8.9 1152 1147 1147 1001 S piperd 0xffffff00044182d8 awk 1151 1147 1147 1001 S wait 0xffffff0004356460 sh 1150 1149 1150 1001 Ss+ select 0xffffff0099e972c0 top 1149 1141 1141 1001 S select 0xffffff0099e9fac0 sshd 1148 1146 1148 1001 Ss kqread 0xffffff00047e8600 tail 1147 1145 1147 1001 Ss wait 0xffffff00049558c0 sh 1146 1140 1140 1001 S select 0xffffff00044683c0 sshd 1145 1139 1139 1001 S select 0xffffff0099e97440 sshd 1141 1004 1141 0 Ss sbwait 0xffffff00046e1e8c sshd 1140 1004 1140 0 Ss sbwait 0xffffff0004940e8c sshd 1139 1004 1139 0 Ss sbwait 0xffffff0004551be4 sshd 1134 1035 1134 1001 S+ wait 0xffffff000469c000 sh 1128 1 1128 0 Ss+ ttyin 0xffffff000423a8a8 getty 1127 1 1127 0 Ss+ ttyin 0xffffff000423aca8 getty 1126 1 1126 0 Ss+ ttyin 0xffffff000423b0a8 getty 1125 1 1125 0 Ss+ ttyin 0xffffff000423b4a8 getty 1124 1 1124 0 Ss+ ttyin 0xffffff000422d4a8 getty 1123 1 1123 0 Ss+ ttyin 0xffffff000422eca8 getty 1122 1 1122 0 Ss+ ttyin 0xffffff0004224ca8 getty 1121 1 1121 0 Ss+ ttyin 0xffffff000422d8a8 getty 1120 1 1120 0 Ss+ ttyin 0xffffff000421dca8 getty 1094 1 1094 0 Ss select 0xffffff00046e5240 inetd 1035 1034 1035 1001 Ss+ wait 0xffffff0004533000 bash 1034 1032 1032 1001 S select 0xffffff0004373240 sshd 1032 1004 1032 0 Ss sbwait 0xffffff0004551694 sshd 1021 1 1021 0 Ss nanslp 0xffffffff80c23668 cron 1015 1 1015 25 Ss pause 0xffffff0004531960 sendmail 1011 1 1011 0 Ss select 0xffffff00042dbc40 sendmail 1004 1 1004 0 Ss select 0xffffff00043d3c40 sshd 967 1 967 0 Ss select 0xffffff00044143c0 ntpd 852 1 852 0 Ss select 0xffffff0004374840 moused 734 1 734 0 Ss select 0xffffff00042db240 syslogd 600 1 600 0 Ss select 0xffffff00043d3140 devd 44 0 0 0 SL flowclea 0xffffffff80c23344 [flowcleaner] 43 0 0 0 SL sdflush 0xffffffff80df3dd8 [softdepflush] 42 0 0 0 SL vlruwt 0xffffff0004357460 [vnlru] 41 0 0 0 SL syncer 0xffffffff80de4c40 [syncer] 40 0 0 0 SL psleep 0xffffffff80de4768 [bufdaemon] 39 0 0 0 SL pgzero 0xffffffff80df580c [pagezero] 38 0 0 0 SL psleep 0xffffffff80df4ba8 [vmdaemon] 9 0 0 0 SL psleep 0xffffffff80df4b6c [pagedaemon] 37 0 0 0 SL wmsg 0xffffff80003e8dd0 [usbus5] 36 0 0 0 SL wmsg 0xffffff80003e8d78 [usbus5] 35 0 0 0 SL wmsg 0xffffff80003e8d20 [usbus5] 34 0 0 0 SL wmsg 0xffffff80003e8cc8 [usbus5] 33 0 0 0 SL wmsg 0xffffff80003e0460 [usbus4] 32 0 0 0 SL wmsg 0xffffff80003e0408 [usbus4] 31 0 0 0 SL wmsg 0xffffff80003e03b0 [usbus4] 30 0 0 0 SL wmsg 0xffffff80003e0358 [usbus4] 29 0 0 0 SL wmsg 0xffffff80003dd460 [usbus3] 28 0 0 0 SL wmsg 0xffffff80003dd408 [usbus3] 27 0 0 0 SL wmsg 0xffffff80003dd3b0 [usbus3] 26 0 0 0 SL wmsg 0xffffff80003dd358 [usbus3] 25 0 0 0 SL wmsg 0xffffff80003da460 [usbus2] 24 0 0 0 SL wmsg 0xffffff80003da408 [usbus2] 23 0 0 0 SL wmsg 0xffffff80003da3b0 [usbus2] 22 0 0 0 SL wmsg 0xffffff80003da358 [usbus2] 21 0 0 0 SL wmsg 0xffffff80003d7460 [usbus1] 20 0 0 0 SL wmsg 0xffffff80003d7408 [usbus1] 19 0 0 0 SL wmsg 0xffffff80003d73b0 [usbus1] 18 0 0 0 SL wmsg 0xffffff80003d7358 [usbus1] 17 0 0 0 SL wmsg 0xffffff80003d4460 [usbus0] 16 0 0 0 SL wmsg 0xffffff80003d4408 [usbus0] 15 0 0 0 SL wmsg 0xffffff80003d43b0 [usbus0] 14 0 0 0 SL wmsg 0xffffff80003d4358 [usbus0] 8 0 0 0 SL waiting_ 0xffffffff80de7b60 [sctp_iterator] 7 0 0 0 SL - 0xffffff0003f0da48 [fdc0] 6 0 0 0 SL - 0xffffff80003eb000 [fw0_probe] 5 0 0 0 SL ccb_scan 0xffffffff80bec5e0 [xpt_thrd] 13 0 0 0 SL - 0xffffffff80c23344 [yarrow] 4 0 0 0 SL - 0xffffffff80c1fb28 [g_down] 3 0 0 0 SL - 0xffffffff80c1fb20 [g_up] 2 0 0 0 SL - 0xffffffff80c1fb10 [g_event] 12 0 0 0 RL (threaded) intr 100042 I [irq12: psm0] 100041 I [irq1: atkbd0] 100039 I [swi0: uart] 100036 I [irq20: fwohci0] 100035 I [irq14: ata0] 100034 I [irq19: ehci0] 100033 I [irq18: ohci2 ohci4] 100032 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: atapci0] 100029 I [irq256: hdac0] 100028 I [irq9: acpi0] 100027 I [swi2: cambio] 100025 I [swi6: task queue] 100024 I [swi6: Giant taskq] 100022 I [swi5: +] 100012 I [swi1: netisr 0] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 Run CPU 2 [swi4: clock] 100008 Run CPU 0 [swi4: clock] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) idle 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 CanRun [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00024e08c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80df3130 [audit] 0 0 0 0 SLs (threaded) kernel 100037 D - 0xffffff0004050b80 [fw0_taskq] 100023 D - 0xffffff000260e980 [thread taskq] 100021 D - 0xffffff0002646080 [acpi_task_2] 100020 D - 0xffffff0002646080 [acpi_task_1] 100019 D - 0xffffff0002646080 [acpi_task_0] 100018 D - 0xffffff0002646100 [kqueue taskq] 100016 D - 0xffffff00024ddd80 [firmware taskq] 100000 D sched 0xffffffff80c1fc20 [swapper] db:0:ps> allt Tracing command gcc pid 1519 tid 100110 td 0xffffff0004465000 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b pmap_release() at pmap_release+0xb0 vmspace_exit() at vmspace_exit+0xcf exit1() at exit1+0x376 sys_exit() at sys_exit+0xe syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x42397c, rsp = 0x7fffffffe7b8, rbp = 0x7fffffffe7f0 --- Tracing command sleep pid 1518 tid 100169 td 0xffffff009d0f9720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071b1bc, rsp = 0x7fffffffebd8, rbp = 0x7fffffffec30 --- Tracing command perl5.8.9 pid 1514 tid 100127 td 0xffffff0004545000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800c20d4c, rsp = 0x7fffffffe618, rbp = 0x740808 --- Tracing command make pid 1492 tid 100171 td 0xffffff009d0f9000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f7fc, rsp = 0x7fffffffd9f8, rbp = 0x7fffffffda40 --- Tracing command sh pid 1465 tid 100135 td 0xffffff000492e720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe288, rbp = 0x46e --- Tracing command make pid 1246 tid 100094 td 0xffffff0004351720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f7fc, rsp = 0x7fffffffda28, rbp = 0x7fffffffda70 --- Tracing command perl5.8.9 pid 1243 tid 100121 td 0xffffff0004494720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800c20d4c, rsp = 0x7fffffffe818, rbp = 0x608808 --- Tracing command awk pid 1152 tid 100149 td 0xffffff0004950ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800874fdc, rsp = 0x7fffffffe8a8, rbp = 0x80088b620 --- Tracing command sh pid 1151 tid 100083 td 0xffffff000434f390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe6a8, rbp = 0x47b --- Tracing command top pid 1150 tid 100099 td 0xffffff000452a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800ab5f5c, rsp = 0x7fffffffea18, rbp = 0xe --- Tracing command sshd pid 1149 tid 100117 td 0xffffff00046e6720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command tail pid 1148 tid 100114 td 0xffffff0004464000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32a kevent() at kevent+0x90 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80072026c, rsp = 0x7fffffffea38, rbp = 0x800802148 --- Tracing command sh pid 1147 tid 100143 td 0xffffff000495dab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffea08, rbp = 0x47b --- Tracing command sshd pid 1146 tid 100113 td 0xffffff0004464390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1145 tid 100118 td 0xffffff00046e6390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1141 tid 100111 td 0xffffff0004464ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d4fdc, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command sshd pid 1140 tid 100084 td 0xffffff00043d8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d4fdc, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command sshd pid 1139 tid 100142 td 0xffffff0004888000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d4fdc, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command sh pid 1134 tid 100119 td 0xffffff00046e6000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe598, rbp = 0x46e --- Tracing command getty pid 1128 tid 100159 td 0xffffff0004893000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1127 tid 100158 td 0xffffff0004893390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1126 tid 100157 td 0xffffff0004893720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1125 tid 100134 td 0xffffff000492eab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1124 tid 100156 td 0xffffff0004893ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1123 tid 100128 td 0xffffff00044a4ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1122 tid 100152 td 0xffffff0004950000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1121 tid 100151 td 0xffffff0004950390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1120 tid 100150 td 0xffffff0004950720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084cfdc, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command inetd pid 1094 tid 100132 td 0xffffff000492f000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a61f5c, rsp = 0x7fffffffdda8, rbp = 0x7878787878787878 --- Tracing command bash pid 1035 tid 100093 td 0xffffff0004351ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b8cd4c, rsp = 0x7fffffffe8e8, rbp = 0x800dc8780 --- Tracing command sshd pid 1034 tid 100125 td 0xffffff0004545720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1032 tid 100092 td 0xffffff00044a4390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d4fdc, rsp = 0x7fffffffdcd8, rbp = 0 --- Tracing command cron pid 1021 tid 100103 td 0xffffff0004529720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093b1bc, rsp = 0x7fffffffeb28, rbp = 0x3b --- Tracing command sendmail pid 1015 tid 100097 td 0xffffff000452b000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3978c, rsp = 0x7fffffffcd68, rbp = 0x1 --- Tracing command sendmail pid 1011 tid 100096 td 0xffffff000452b390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd4f5c, rsp = 0x7fffffffc1e8, rbp = 0x7fffffffc280 --- Tracing command sshd pid 1004 tid 100095 td 0xffffff0004351390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffddd8, rbp = 0x1 --- Tracing command ntpd pid 967 tid 100107 td 0xffffff0004465ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800d4ef5c, rsp = 0x7fffffffec18, rbp = 0x7fffffffed48 --- Tracing command moused pid 852 tid 100105 td 0xffffff0004529000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80096ef5c, rsp = 0x7fffffffe878, rbp = 0x7fffffffed40 --- Tracing command syslogd pid 734 tid 100100 td 0xffffff000452a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80084ff5c, rsp = 0x7fffffffdcb8, rbp = 0x5 --- Tracing command devd pid 600 tid 100088 td 0xffffff00043d8000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x4410ac, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 --- Tracing command flowcleaner pid 44 tid 100075 td 0xffffff000434f720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 flowtable_cleaner() at flowtable_cleaner+0x139 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736bed30, rbp = 0 --- Tracing command softdepflush pid 43 tid 100074 td 0xffffff000434fab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736b9d30, rbp = 0 --- Tracing command vnlru pid 42 tid 100073 td 0xffffff0004350000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vnlru_proc() at vnlru_proc+0x608 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736b4d30, rbp = 0 --- Tracing command syncer pid 41 tid 100072 td 0xffffff0004350390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4dc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736afd30, rbp = 0 --- Tracing command bufdaemon pid 40 tid 100071 td 0xffffff0004350720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736aad30, rbp = 0 --- Tracing command pagezero pid 39 tid 100070 td 0xffffff0004350ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736a5d30, rbp = 0 --- Tracing command vmdaemon pid 38 tid 100069 td 0xffffff0004351000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736a0d30, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100068 td 0xffffff00042b7000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807369bd30, rbp = 0 --- Tracing command usbus5 pid 37 tid 100067 td 0xffffff00042b7390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073636d30, rbp = 0 --- Tracing command usbus5 pid 36 tid 100066 td 0xffffff00042b7720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073631d30, rbp = 0 --- Tracing command usbus5 pid 35 tid 100065 td 0xffffff00042b7ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807362cd30, rbp = 0 --- Tracing command usbus5 pid 34 tid 100064 td 0xffffff00042b8000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073627d30, rbp = 0 --- Tracing command usbus4 pid 33 tid 100063 td 0xffffff00042b8390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073622d30, rbp = 0 --- Tracing command usbus4 pid 32 tid 100062 td 0xffffff00042b8720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807361dd30, rbp = 0 --- Tracing command usbus4 pid 31 tid 100061 td 0xffffff00042b8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073618d30, rbp = 0 --- Tracing command usbus4 pid 30 tid 100060 td 0xffffff00042ba000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073613d30, rbp = 0 --- Tracing command usbus3 pid 29 tid 100059 td 0xffffff00042ba390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807360ed30, rbp = 0 --- Tracing command usbus3 pid 28 tid 100058 td 0xffffff00042ba720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073609d30, rbp = 0 --- Tracing command usbus3 pid 27 tid 100057 td 0xffffff00042baab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073604d30, rbp = 0 --- Tracing command usbus3 pid 26 tid 100056 td 0xffffff0003f12390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735ffd30, rbp = 0 --- Tracing command usbus2 pid 25 tid 100055 td 0xffffff0003f12720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735fad30, rbp = 0 --- Tracing command usbus2 pid 24 tid 100054 td 0xffffff0003f12ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735f5d30, rbp = 0 --- Tracing command usbus2 pid 23 tid 100053 td 0xffffff0004296000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735f0d30, rbp = 0 --- Tracing command usbus2 pid 22 tid 100052 td 0xffffff0004296390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735ebd30, rbp = 0 --- Tracing command usbus1 pid 21 tid 100051 td 0xffffff0004296720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735e6d30, rbp = 0 --- Tracing command usbus1 pid 20 tid 100050 td 0xffffff0004296ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735e1d30, rbp = 0 --- Tracing command usbus1 pid 19 tid 100049 td 0xffffff000429a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735dcd30, rbp = 0 --- Tracing command usbus1 pid 18 tid 100048 td 0xffffff000429a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735d7d30, rbp = 0 --- Tracing command usbus0 pid 17 tid 100047 td 0xffffff000429a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735d2d30, rbp = 0 --- Tracing command usbus0 pid 16 tid 100046 td 0xffffff000429aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735cdd30, rbp = 0 --- Tracing command usbus0 pid 15 tid 100045 td 0xffffff0002663ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735c8d30, rbp = 0 --- Tracing command usbus0 pid 14 tid 100044 td 0xffffff0003f0f000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735c3d30, rbp = 0 --- Tracing command sctp_iterator pid 8 tid 100043 td 0xffffff0003f0f390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 sctp_iterator_thread() at sctp_iterator_thread+0x54 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735bed30, rbp = 0 --- Tracing command fdc0 pid 7 tid 100040 td 0xffffff0003f10000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735acd30, rbp = 0 --- Tracing command fw0_probe pid 6 tid 100038 td 0xffffff0003f10720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 fw_bus_probe_thread() at fw_bus_probe_thread+0xbc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073594d30, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100026 td 0xffffff00024f4720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a8d30, rbp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xffffff00024f3000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800007bd30, rbp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xffffff00024f3720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000071d30, rbp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xffffff00024f3ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800006cd30, rbp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xffffff00024f4000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000067d30, rbp = 0 --- Tracing command intr pid 12 tid 100042 td 0xffffff0003f0f720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xffffff0003f0fab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735b1d30, rbp = 0 --- Tracing command intr pid 12 tid 100039 td 0xffffff0003f10390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807359dd30, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xffffff0003f12000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073340d30, rbp = 0 --- Tracing command intr pid 12 tid 100035 td 0xffffff000265e720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807333bd30, rbp = 0 --- Tracing command intr pid 12 tid 100034 td 0xffffff000265eab0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xffffff0002661000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80730bfd30, rbp = 0 --- Tracing command intr pid 12 tid 100032 td 0xffffff0002661390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073076d30, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xffffff0002661720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807302dd30, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xffffff0002661ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80729dcd30, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff0002663000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b7d30, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xffffff0002663390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b2d30, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xffffff0002663720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000add30, rbp = 0 --- Tracing command intr pid 12 tid 100025 td 0xffffff00024f4ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a3d30, rbp = 0 --- Tracing command intr pid 12 tid 100024 td 0xffffff000265c000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100022 td 0xffffff000265c720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000094d30, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00024f4390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000062d30, rbp = 0 --- Tracing command intr pid 12 tid 100011 td 0xffffff00024e4390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800005dd30, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xffffff00024e4720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000058d30, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xffffff00024e4ab0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff801fcf16, rsp = 0xffffff800001efe0, rbp = 0xffffff8000053aa0 --- acpi_timer_read() at acpi_timer_read+0x16 acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x14 binuptime() at binuptime+0x3b softclock() at softclock+0x267 intr_event_execute_handlers() at intr_event_execute_handlers+0x68 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000053d30, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xffffff00024ef000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff801fcf16, rsp = 0xffffffff80e36140, rbp = 0xffffff800004eaa0 --- acpi_timer_read() at acpi_timer_read+0x16 acpi_timer_get_timecount_safe() at acpi_timer_get_timecount_safe+0x14 binuptime() at binuptime+0x3b softclock() at softclock+0x267 intr_event_execute_handlers() at intr_event_execute_handlers+0x68 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800004ed30, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xffffff00024ef390 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xffffff00024ef720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_idletd() at sched_idletd+0xf6 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff00024e3000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff80844bd6, rsp = 0xffffff800003fb50, rbp = 0xffffff800003fb60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800003fd30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00024e3390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff80844bd6, rsp = 0xffffff800003ab50, rbp = 0xffffff800003ab60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800003ad30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff00024e3720 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80844bd6, rsp = 0xffffff8000025fe0, rbp = 0xffffff8000035b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000035d30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00024e3ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c57c, rsp = 0x7fffffffe808, rbp = 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff00024e4000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800002bd30, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xffffff0003f10ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80734cfd30, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xffffff000265c390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000099d30, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xffffff000265cab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800008fd30, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xffffff000265e000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800008ad30, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xffffff000265e390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000085d30, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xffffff00024efab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000080d30, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00024f3390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000076d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c20080 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x292 mi_startup() at mi_startup+0x59 btext() at btext+0x2c db:0:allt> call doadump Physical memory: 3951 MB Dumping 1498 MB: 1483 1467 1451 1435 1419 1403 1387 1371 1355 1339 1323 1307 1291 1275 1259 1243 1227 1211 1195 1179 1163 1147 1131 1115 1099 1083 1067 1051 1035 1019 1003 987 971 955 939 923 907 891 875 859 843 827 811 795 779 763 747 731 715 699 683 667 651 635 619 603 587 571 555 539 523 507 491 475 459 443 427 411 395 379 363 347 331 315 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11 Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 (kgdb) bt #0 doadump () at pcpu.h:223 #1 0xffffffff801d92ac in db_fncall (dummy1=Variable "dummy1" is not available. ) at ../../../ddb/db_command.c:548 #2 0xffffffff801d955d in db_command (last_cmdp=0xffffffff80bedc20, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #3 0xffffffff801ddda3 in db_script_exec (scriptname=0xffffffff80bee4c0 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff801d95e1 in db_command (last_cmdp=0xffffffff80bedc20, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #5 0xffffffff801d9830 in db_command_loop () at ../../../ddb/db_command.c:498 #6 0xffffffff801db809 in db_trap (type=Variable "type" is not available. ) at ../../../ddb/db_main.c:229 #7 0xffffffff805b3d45 in kdb_trap (type=0x3, code=0x0, tf=0xffffff8075a70890) at ../../../kern/subr_kdb.c:535 #8 0xffffffff80867541 in trap (frame=0xffffff8075a70890) at ../../../amd64/amd64/trap.c:611 #9 0xffffffff8084d153 in calltrap () at ../../../amd64/amd64/exception.S:224 #10 0xffffffff805b3f1d in kdb_enter (why=0xffffffff80960881 "panic", msg=0xa
) at cpufunc.h:63 #11 0xffffffff805845bb in panic (fmt=Variable "fmt" is not available. ) at ../../../kern/kern_shutdown.c:562 #12 0xffffffff8085e3e0 in pmap_release (pmap=0xffffff8075a707c0) at ../../../amd64/amd64/pmap.c:1658 #13 0xffffffff807d0b2f in vmspace_exit (td=0xffffff0004465000) at ../../../vm/vm_map.c:333 #14 0xffffffff8055a7f6 in exit1 (td=0xffffff0004465000, rv=0x200) at ../../../kern/kern_exit.c:298 #15 0xffffffff8055b4ae in sys_exit (td=Variable "td" is not available. ) at ../../../kern/kern_exit.c:109 #16 0xffffffff80866ef0 in syscall (frame=0xffffff8075a70c80) at ../../../amd64/amd64/trap.c:982 #17 0xffffffff8084d431 in Xfast_syscall () at ../../../amd64/amd64/exception.S:373 #18 0x000000000042397c in ?? () (kgdb) f 12 #12 0xffffffff8085e3e0 in pmap_release (pmap=0xffffff8075a707c0) at ../../../amd64/amd64/pmap.c:1658 1658 KASSERT(pmap->pm_stats.resident_count == 0, (kgdb) l 1653 void 1654 pmap_release(pmap_t pmap) 1655 { 1656 vm_page_t m; 1657 1658 KASSERT(pmap->pm_stats.resident_count == 0, 1659 ("pmap_release: pmap resident count %ld != 0", 1660 pmap->pm_stats.resident_count)); 1661 KASSERT(pmap->pm_root == NULL, 1662 ("pmap_release: pmap has reserved page table page(s)")); (kgdb) p *pmap $3 = {pm_mtx = {lock_object = {lo_name = 0xffffff00044de000 "`¤\022\235", lo_flags = 0x203a7265, lo_data = 0x696e6170, lo_witness = 0xffffff80000a0d63}, mtx_lock = 0x80}, pm_pml4 = 0x203d206469757000, pm_pvchunk = {tqh_first = 0x656c6572000a0d31, tqh_last = 0x616d70203a657361}, pm_active = 0x65722070, pm_stats = {resident_count = 0x746e756f6320746e, wired_count = 0x30203d2120312d20}, pm_root = 0xffffff8075000a0d} (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/vm/vm_map.c =================================================================== --- /usr/src/sys/vm/vm_map.c (revision 196259) +++ /usr/src/sys/vm/vm_map.c (working copy) @@ -313,6 +313,7 @@ vm_init2(void) static inline void vmspace_dofree(struct vmspace *vm) { + CTR1(KTR_VM, "vmspace_free: %p", vm); /* @@ -329,12 +330,8 @@ vmspace_dofree(struct vmspace *vm) (void)vm_map_remove(&vm->vm_map, vm->vm_map.min_offset, vm->vm_map.max_offset); - /* - * XXX Comment out the pmap_release call for now. The - * vmspace_zone is marked as UMA_ZONE_NOFREE, and bugs cause - * pmap.resident_count to be != 0 on exit sometimes. - */ -/* pmap_release(vmspace_pmap(vm)); */ + pmap_release(vmspace_pmap(vm)); + vm->vm_map.pmap = NULL; uma_zfree(vmspace_zone, vm); } Index: /usr/src/sys/amd64/amd64/pmap.c =================================================================== --- /usr/src/sys/amd64/amd64/pmap.c (revision 196259) +++ /usr/src/sys/amd64/amd64/pmap.c (working copy) @@ -2453,8 +2453,8 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offse /* * Perform an unsynchronized read. This is, however, safe. */ - if (pmap->pm_stats.resident_count == 0) - return; +// if (pmap->pm_stats.resident_count == 0) +// return; anyvalid = 0; @@ -2476,8 +2476,8 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offse for (; sva < eva; sva = va_next) { - if (pmap->pm_stats.resident_count == 0) - break; +// if (pmap->pm_stats.resident_count == 0) +// break; pml4e = pmap_pml4e(pmap, sva); if ((*pml4e & PG_V) == 0) { -- Test scenario: by