GENERIC HEAD + ZERO_COPY_SOCKETS from 2009-08-12 16:27:51 UTC, r196149, vmcore.8 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-BETA2 #0 r196149: Wed Aug 12 20:16:34 CEST 2009 pho@x4.osted.lan:/usr/src/sys/amd64/compile/ZERO WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3974782976 (3790 MB) : Trying to mount root from ufs:/dev/ad4s2a Entropy harvesting: interrupts ethernet point_to_point kickstart. /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 637978 free (906 frags, 79634 blocks, 0.1% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 50692457 free (705 frags, 6336469 blocks, 0.0% fragmentation) /dev/ad4s2f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2f: clean, 717657 free (136017 frags, 72705 blocks, 2.7% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 46064397 free (46901 frags, 5752187 blocks, 0.1% fragmentation) Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. add net default: gateway 192.168.1.1 Configuring keyboard: keymap. Configuring syscons: keymap blanktime. Wed Aug 12 20:26:59 CEST 2009 FreeBSD/amd64 (x4.osted.lan) (console) login: Expensive timeout(9) function: 0xffffffff8046e150(0xffffffff80e4f0e0) 0.012044265 s lock order reversal: 1st 0xffffff804fc20e78 bufwait (bufwait) @ kern/vfs_bio.c:2559 2nd 0xffffff00044f8c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x5e4 ufs_makeinode() at ufs_makeinode+0x2d5 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800c74dfc, rsp = 0x7fffffffe258, rbp = 0xa --- ugen2.2: at usbus2 (disconnected) ukbd0: at uhub5, port 2, addr 2 (disconnected) ums0: at uhub5, port 2, addr 2 (disconnected) lock order reversal: 1st 0xffffff0004971578 ufs (ufs) @ kern/vfs_lookup.c:497 2nd 0xffffff8050677f58 bufwait (bufwait) @ ufs/ffs/ffs_softdep.c:6170 3rd 0xffffff0039b65a58 ufs (ufs) @ kern/vfs_subr.c:2083 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd03 ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b vfs_hash_get() at vfs_hash_get+0xd5 ffs_vgetf() at ffs_vgetf+0x49 softdep_sync_metadata() at softdep_sync_metadata+0x456 ffs_syncvnode() at ffs_syncvnode+0x210 ffs_truncate() at ffs_truncate+0x19c ufs_direnter() at ufs_direnter+0x44f ufs_mkdir() at ufs_mkdir+0x792 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (136, FreeBSD ELF32, mkdir), rip = 0x2817b353, rsp = 0xffffd5ec, rbp = 0xffffd688 --- panic: vm_page_free_toq: freeing mapped page 0xffffff00d50e28c0 cpuid = 0 KDB: enter: panic [thread pid 35004 tid 100186 ] Stopped at kdb_enter+0x3d: movq $0,0x6902f0(%rip) db> set $lines 0 db> bt Tracing pid 35004 tid 100186 td 0xffffff0004e70720 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b vm_page_free_toq() at vm_page_free_toq+0x206 vm_object_terminate() at vm_object_terminate+0xb7 vm_object_deallocate() at vm_object_deallocate+0x17a _vm_map_unlock() at _vm_map_unlock+0x70 vm_map_remove() at vm_map_remove+0x6f vmspace_free() at vmspace_free+0x56 vmspace_exec() at vmspace_exec+0x56 exec_new_vmspace() at exec_new_vmspace+0x133 exec_elf64_imgact() at exec_elf64_imgact+0x2e6 kern_execve() at kern_execve+0x3b2 freebsd32_execve() at freebsd32_execve+0x3d ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (59, FreeBSD ELF32, freebsd32_execve), rip = 0x280fe1fb, rsp = 0xffffcdec, rbp = 0xffffce48 --- db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x4d8800 curthread = 0xffffff0004e70720: pid 35004 "run" curpcb = 0xffffff8075c4ad40 fpcurthread = none idlethread = 0xffffff00024ef720: pid 11 "idle: cpu0" curpmap = 0 tssp = 0xffffffff80e2f080 commontssp = 0xffffffff80e2f080 rsp0 = 0xffffff8075c4ad40 gs32p = 0xffffffff80e2deb8 ldt = 0xffffffff80e2def8 tss = 0xffffffff80e2dee8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f43b800 curthread = 0xffffff00024e3000: pid 11 "idle: cpu1" curpcb = 0xffffff800003fd40 fpcurthread = none idlethread = 0xffffff00024e3000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff80e2f0e8 commontssp = 0xffffffff80e2f0e8 rsp0 = 0xffffff800003fd40 gs32p = 0xffffffff80e2df20 ldt = 0xffffffff80e2df60 tss = 0xffffffff80e2df50 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f442800 curthread = 0xffffff00024e3390: pid 11 "idle: cpu2" curpcb = 0xffffff800003ad40 fpcurthread = none idlethread = 0xffffff00024e3390: pid 11 "idle: cpu2" curpmap = 0 tssp = 0xffffffff80e2f150 commontssp = 0xffffffff80e2f150 rsp0 = 0xffffff800003ad40 gs32p = 0xffffffff80e2df88 ldt = 0xffffffff80e2dfc8 tss = 0xffffffff80e2dfb8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f449800 curthread = 0xffffff00024e3720: pid 11 "idle: cpu3" curpcb = 0xffffff8000035d40 fpcurthread = none idlethread = 0xffffff00024e3720: pid 11 "idle: cpu3" curpmap = 0 tssp = 0xffffffff80e2f1b8 commontssp = 0xffffffff80e2f1b8 rsp0 = 0xffffff8000035d40 gs32p = 0xffffffff80e2dff0 ldt = 0xffffffff80e2e030 tss = 0xffffffff80e2e020 spin locks held: db> show alllocks Process 35004 (run) thread 0xffffff0004e70720 (100186) exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff80df57e0) locked @ vm/vm_object.c:720 exclusive sleep mutex vm object (standard object) r = 0 (0xffffff00be2f4360) locked @ vm/vm_object.c:482 Process 1140 (sshd) thread 0xffffff0004b7e000 (100148) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004545648) locked @ kern/uipc_sockbuf.c:148 Process 1139 (sshd) thread 0xffffff0004b7e390 (100147) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004c2f0f8) locked @ kern/uipc_sockbuf.c:148 Process 1137 (sshd) thread 0xffffff00042cf000 (100145) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004c2fb98) locked @ kern/uipc_sockbuf.c:148 Process 1123 (sshd) thread 0xffffff000441b000 (100078) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004539648) locked @ kern/uipc_sockbuf.c:148 Process 1115 (sshd) thread 0xffffff00044fc390 (100110) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004737648) locked @ kern/uipc_sockbuf.c:148 db> show lockedvnods Locked vnodes db> ps pid ppid pgrp uid state wmesg wchan cmd 35004 34871 34831 1001 R+ CPU 0 run 34999 1149 1144 1001 S nanslp 0xffffffff80c242e8 sleep 34871 34870 34831 1001 S+ wait 0xffffff0039083000 run 34870 34869 34831 1001 S+ wait 0xffffff00392f28c0 run 34869 34844 34831 1001 S+ nanslp 0xffffffff80c242e8 run 34844 34831 34831 1001 S+ wait 0xffffff000479c460 sh 34831 1118 34831 1001 S+ wait 0xffffff00044108c0 sh 1150 1144 1144 1001 S piperd 0xffffff00044ac2d8 awk 1149 1144 1144 1001 S wait 0xffffff0004527460 sh 1148 1146 1148 1001 Ss+ select 0xffffff0004367940 top 1147 1145 1147 1001 Ss kqread 0xffffff000482de00 tail 1146 1140 1140 1001 S select 0xffffff0004b511c0 sshd 1145 1139 1139 1001 S select 0xffffff0004466cc0 sshd 1144 1143 1144 1001 Ss wait 0xffffff0004b548c0 sh 1143 1137 1137 1001 S select 0xffffff0004af24c0 sshd 1140 1033 1140 0 Ss sbwait 0xffffff0004545694 sshd 1139 1033 1139 0 Ss sbwait 0xffffff0004c2f144 sshd 1137 1033 1137 0 Ss sbwait 0xffffff0004c2fbe4 sshd 1126 1125 1126 1001 Ss+ ttyin 0xffffff0004194ca8 bash 1125 1123 1123 1001 S select 0xffffff000471b140 sshd 1123 1033 1123 0 Ss sbwait 0xffffff0004539694 sshd 1118 1117 1118 1001 Ss+ wait 0xffffff000440e8c0 bash 1117 1115 1115 1001 S select 0xffffff000471b1c0 sshd 1115 1033 1115 0 Ss sbwait 0xffffff0004737694 sshd 1114 1 1114 0 Ss+ ttyin 0xffffff00041ba8a8 getty 1113 1 1113 0 Ss+ ttyin 0xffffff00041baca8 getty 1112 1 1112 0 Ss+ ttyin 0xffffff00041bb0a8 getty 1111 1 1111 0 Ss+ ttyin 0xffffff00041bb4a8 getty 1110 1 1110 0 Ss+ ttyin 0xffffff00041ad4a8 getty 1109 1 1109 0 Ss+ ttyin 0xffffff00041aeca8 getty 1108 1 1108 0 Ss+ ttyin 0xffffff00041a4ca8 getty 1107 1 1107 0 Ss+ ttyin 0xffffff00041ad8a8 getty 1106 1 1106 0 Ss+ ttyin 0xffffff000419dca8 getty 1080 1 1080 0 Ss select 0xffffff000471b740 inetd 1050 1 1050 0 Ss nanslp 0xffffffff80c242e8 cron 1044 1 1044 25 Ss pause 0xffffff0004519500 sendmail 1040 1 1040 0 Ss select 0xffffff000471bbc0 sendmail 1033 1 1033 0 Ss select 0xffffff00047381c0 sshd 996 1 996 0 Ss select 0xffffff00044678c0 ntpd 881 1 881 0 Ss select 0xffffff00047388c0 moused 763 1 763 0 Ss select 0xffffff00044c7940 syslogd 629 1 629 0 Ss select 0xffffff00044c7b40 devd 44 0 0 0 SL flowclea 0xffffffff80c23fc4 [flowcleaner] 43 0 0 0 SL sdflush 0xffffffff80df4a98 [softdepflush] 42 0 0 0 SL syncer 0xffffffff80de5900 [syncer] 41 0 0 0 SL vlruwt 0xffffff00042d88c0 [vnlru] 40 0 0 0 SL psleep 0xffffffff80de5428 [bufdaemon] 39 0 0 0 SL pgzero 0xffffffff80df64cc [pagezero] 38 0 0 0 SL psleep 0xffffffff80df5868 [vmdaemon] 9 0 0 0 SL psleep 0xffffffff80df582c [pagedaemon] 37 0 0 0 SL wmsg 0xffffff80003e8dd0 [usbus5] 36 0 0 0 SL wmsg 0xffffff80003e8d78 [usbus5] 35 0 0 0 SL wmsg 0xffffff80003e8d20 [usbus5] 34 0 0 0 SL wmsg 0xffffff80003e8cc8 [usbus5] 33 0 0 0 SL wmsg 0xffffff80003e0460 [usbus4] 32 0 0 0 SL wmsg 0xffffff80003e0408 [usbus4] 31 0 0 0 SL wmsg 0xffffff80003e03b0 [usbus4] 30 0 0 0 SL wmsg 0xffffff80003e0358 [usbus4] 29 0 0 0 SL wmsg 0xffffff80003dd460 [usbus3] 28 0 0 0 SL wmsg 0xffffff80003dd408 [usbus3] 27 0 0 0 SL wmsg 0xffffff80003dd3b0 [usbus3] 26 0 0 0 SL wmsg 0xffffff80003dd358 [usbus3] 25 0 0 0 SL wmsg 0xffffff80003da460 [usbus2] 24 0 0 0 SL wmsg 0xffffff80003da408 [usbus2] 23 0 0 0 SL wmsg 0xffffff80003da3b0 [usbus2] 22 0 0 0 SL wmsg 0xffffff80003da358 [usbus2] 21 0 0 0 SL wmsg 0xffffff80003d7460 [usbus1] 20 0 0 0 SL wmsg 0xffffff80003d7408 [usbus1] 19 0 0 0 SL wmsg 0xffffff80003d73b0 [usbus1] 18 0 0 0 SL wmsg 0xffffff80003d7358 [usbus1] 17 0 0 0 SL wmsg 0xffffff80003d4460 [usbus0] 16 0 0 0 SL wmsg 0xffffff80003d4408 [usbus0] 15 0 0 0 SL wmsg 0xffffff80003d43b0 [usbus0] 14 0 0 0 SL wmsg 0xffffff80003d4358 [usbus0] 8 0 0 0 SL waiting_ 0xffffffff80de8820 [sctp_iterator] 7 0 0 0 SL - 0xffffff0003fe5a48 [fdc0] 6 0 0 0 SL - 0xffffff80003eb000 [fw0_probe] 5 0 0 0 SL ccb_scan 0xffffffff80bed260 [xpt_thrd] 13 0 0 0 SL - 0xffffffff80c23fc4 [yarrow] 4 0 0 0 SL - 0xffffffff80c207a8 [g_down] 3 0 0 0 SL - 0xffffffff80c207a0 [g_up] 2 0 0 0 SL - 0xffffffff80c20790 [g_event] 12 0 0 0 WL (threaded) intr 100042 I [irq12: psm0] 100041 I [irq1: atkbd0] 100039 I [swi0: uart] 100036 I [irq20: fwohci0] 100035 I [irq14: ata0] 100034 I [irq19: ehci0] 100033 I [irq18: ohci2 ohci4] 100032 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: atapci0] 100029 I [irq256: hdac0] 100028 I [irq9: acpi0] 100026 I [swi2: cambio] 100024 I [swi6: task queue] 100023 I [swi6: Giant taskq] 100021 I [swi5: +] 100012 I [swi3: vm] 100011 I [swi1: netisr 0] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi4: clock] 100007 I [swi4: clock] 11 0 0 0 RL (threaded) idle 100006 CanRun [idle: cpu0] 100005 Run CPU 1 [idle: cpu1] 100004 Run CPU 2 [idle: cpu2] 100003 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00024e08c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80df3df0 [audit] 0 0 0 0 SLs (threaded) kernel 100037 D - 0xffffff0004110b80 [fw0_taskq] 100027 D - 0xffffff0002660980 [kqueue taskq] 100022 D - 0xffffff000260ea00 [thread taskq] 100020 D - 0xffffff0002646100 [acpi_task_2] 100019 D - 0xffffff0002646100 [acpi_task_1] 100018 D - 0xffffff0002646100 [acpi_task_0] 100016 D - 0xffffff00024ddd80 [firmware taskq] 100000 D sched 0xffffffff80c208a0 [swapper] db> allt Tracing command run pid 35004 tid 100186 td 0xffffff0004e70720 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b vm_page_free_toq() at vm_page_free_toq+0x206 vm_object_terminate() at vm_object_terminate+0xb7 vm_object_deallocate() at vm_object_deallocate+0x17a _vm_map_unlock() at _vm_map_unlock+0x70 vm_map_remove() at vm_map_remove+0x6f vmspace_free() at vmspace_free+0x56 vmspace_exec() at vmspace_exec+0x56 exec_new_vmspace() at exec_new_vmspace+0x133 exec_elf64_imgact() at exec_elf64_imgact+0x2e6 kern_execve() at kern_execve+0x3b2 freebsd32_execve() at freebsd32_execve+0x3d ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (59, FreeBSD ELF32, freebsd32_execve), rip = 0x280fe1fb, rsp = 0xffffcdec, rbp = 0xffffce48 --- Tracing command sleep pid 34999 tid 100235 td 0xffffff0039bfe390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071b1bc, rsp = 0x7fffffffebd8, rbp = 0x7fffffffec30 --- Tracing command run pid 34871 tid 100205 td 0xffffff003905dab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 freebsd32_wait4() at freebsd32_wait4+0x35 ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (7, FreeBSD ELF32, freebsd32_wait4), rip = 0x280fe23b, rsp = 0xffffce4c, rbp = 0xffffcef8 --- Tracing command run pid 34870 tid 100241 td 0xffffff0039305ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 freebsd32_wait4() at freebsd32_wait4+0x35 ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (7, FreeBSD ELF32, freebsd32_wait4), rip = 0x280fe23b, rsp = 0xffffd43c, rbp = 0xffffd458 --- Tracing command run pid 34869 tid 100187 td 0xffffff0004e70390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 freebsd32_nanosleep() at freebsd32_nanosleep+0x6f ia32_syscall() at ia32_syscall+0x19c Xint0x80_syscall() at Xint0x80_syscall+0x95 --- syscall (240, FreeBSD ELF32, freebsd32_nanosleep), rip = 0x2816b817, rsp = 0xffffd45c, rbp = 0xffffd488 --- Tracing command sh pid 34844 tid 100134 td 0xffffff000441bab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe358, rbp = 0x880f --- Tracing command sh pid 34831 tid 100077 td 0xffffff000441b390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe308, rbp = 0x880f --- Tracing command awk pid 1150 tid 100122 td 0xffffff000469e390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800874fdc, rsp = 0x7fffffffe8a8, rbp = 0x80088b620 --- Tracing command sh pid 1149 tid 100101 td 0xffffff00043c4ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffe6a8, rbp = 0x478 --- Tracing command top pid 1148 tid 100117 td 0xffffff000469f720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800ab5f5c, rsp = 0x7fffffffea18, rbp = 0xe --- Tracing command tail pid 1147 tid 100081 td 0xffffff000441a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32a kevent() at kevent+0x90 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80072026c, rsp = 0x7fffffffea38, rbp = 0x800802148 --- Tracing command sshd pid 1146 tid 100140 td 0xffffff0004843720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1145 tid 100153 td 0xffffff0004b78ab0 Tracing command sh pid 1144 tid 100152 td 0xffffff0004b7b000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935d4c, rsp = 0x7fffffffea08, rbp = 0x478 --- Tracing command sshd pid 1143 tid 100151 td 0xffffff0004b7b390 Tracing command sshd pid 1140 tid 100148 td 0xffffff0004b7e000 Tracing command sshd pid 1139 tid 100147 td 0xffffff0004b7e390 Tracing command sshd pid 1137 tid 100145 td 0xffffff00042cf000 Tracing command bash pid 1126 tid 100093 td 0xffffff0004411720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800c11fdc, rsp = 0x7fffffffe728, rbp = 0x7fffffffe747 --- Tracing command sshd pid 1125 tid 100121 td 0xffffff000469e720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d4f5c, rsp = 0x7fffffffdcb8, rbp = 0x7fffffffdd40 --- Tracing command sshd pid 1123 tid 100078 td 0xffffff000441b000 Tracing command bash pid 1118 tid 100084 td 0xffffff0004419ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b8cd4c, rsp = 0x7fffffffe8e8, rbp = 0x800dc49a0 --- Tracing command sshd pid 1117 tid 100105 td 0xffffff0004522720 Tracing command sshd pid 1115 tid 100110 td 0xffffff00044fc390 Tracing command getty pid 1114 tid 100099 td 0xffffff0004411390 Tracing command getty pid 1113 tid 100112 td 0xffffff00043c3000 Tracing command getty pid 1112 tid 100108 td 0xffffff00044fcab0 Tracing command getty pid 1111 tid 100080 td 0xffffff000441a720 Tracing command getty pid 1110 tid 100083 td 0xffffff00042d0390 Tracing command getty pid 1109 tid 100114 td 0xffffff00043c0720 Tracing command getty pid 1108 tid 100115 td 0xffffff00043c0390 Tracing command getty pid 1107 tid 100116 td 0xffffff00043c0000 Tracing command getty pid 1106 tid 100076 td 0xffffff00043c4720 Tracing command inetd pid 1080 tid 100144 td 0xffffff0004842720 Tracing command cron pid 1050 tid 100131 td 0xffffff00044cd720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093b1bc, rsp = 0x7fffffffeb28, rbp = 0x32 --- Tracing command sendmail pid 1044 tid 100107 td 0xffffff0004522000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3978c, rsp = 0x7fffffffcd68, rbp = 0x1 --- Tracing command sendmail pid 1040 tid 100119 td 0xffffff000469f000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd4f5c, rsp = 0x7fffffffc1e8, rbp = 0x7fffffffc280 --- Tracing command sshd pid 1033 tid 100102 td 0xffffff000451c390 Tracing command ntpd pid 996 tid 100079 td 0xffffff000441aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800d4ef5c, rsp = 0x7fffffffec18, rbp = 0x7fffffffed48 --- Tracing command moused pid 881 tid 100092 td 0xffffff0004411ab0 Tracing command syslogd pid 763 tid 100090 td 0xffffff00042cf720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x5c1 select() at select+0x56 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80084ff5c, rsp = 0x7fffffffdcb8, rbp = 0x5 --- Tracing command devd pid 629 tid 100089 td 0xffffff00042cfab0 Tracing command flowcleaner pid 44 tid 100075 td 0xffffff00042d0720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 flowtable_cleaner() at flowtable_cleaner+0x139 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736bed30, rbp = 0 --- Tracing command softdepflush pid 43 tid 100074 td 0xffffff00042d0ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736b9d30, rbp = 0 --- Tracing command syncer pid 42 tid 100073 td 0xffffff00042d1000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4dc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736b4d30, rbp = 0 --- Tracing command vnlru pid 41 tid 100072 td 0xffffff00042d1390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vnlru_proc() at vnlru_proc+0x608 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736afd30, rbp = 0 --- Tracing command bufdaemon pid 40 tid 100071 td 0xffffff00042d1720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736aad30, rbp = 0 --- Tracing command pagezero pid 39 tid 100070 td 0xffffff00042d1ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736a5d30, rbp = 0 --- Tracing command vmdaemon pid 38 tid 100069 td 0xffffff00042d2000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80736a0d30, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100068 td 0xffffff0004237000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807369bd30, rbp = 0 --- Tracing command usbus5 pid 37 tid 100067 td 0xffffff0004237390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073636d30, rbp = 0 --- Tracing command usbus5 pid 36 tid 100066 td 0xffffff0004237720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073631d30, rbp = 0 --- Tracing command usbus5 pid 35 tid 100065 td 0xffffff0004237ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807362cd30, rbp = 0 --- Tracing command usbus5 pid 34 tid 100064 td 0xffffff0004238000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073627d30, rbp = 0 --- Tracing command usbus4 pid 33 tid 100063 td 0xffffff0004238390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073622d30, rbp = 0 --- Tracing command usbus4 pid 32 tid 100062 td 0xffffff0004238720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807361dd30, rbp = 0 --- Tracing command usbus4 pid 31 tid 100061 td 0xffffff0004238ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073618d30, rbp = 0 --- Tracing command usbus4 pid 30 tid 100060 td 0xffffff000423a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073613d30, rbp = 0 --- Tracing command usbus3 pid 29 tid 100059 td 0xffffff000423a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807360ed30, rbp = 0 --- Tracing command usbus3 pid 28 tid 100058 td 0xffffff000423a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073609d30, rbp = 0 --- Tracing command usbus3 pid 27 tid 100057 td 0xffffff000423aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073604d30, rbp = 0 --- Tracing command usbus3 pid 26 tid 100056 td 0xffffff0003fea390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735ffd30, rbp = 0 --- Tracing command usbus2 pid 25 tid 100055 td 0xffffff0003fea720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735fad30, rbp = 0 --- Tracing command usbus2 pid 24 tid 100054 td 0xffffff0003feaab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735f5d30, rbp = 0 --- Tracing command usbus2 pid 23 tid 100053 td 0xffffff0004216000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735f0d30, rbp = 0 --- Tracing command usbus2 pid 22 tid 100052 td 0xffffff0004216390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735ebd30, rbp = 0 --- Tracing command usbus1 pid 21 tid 100051 td 0xffffff0004216720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735e6d30, rbp = 0 --- Tracing command usbus1 pid 20 tid 100050 td 0xffffff0004216ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735e1d30, rbp = 0 --- Tracing command usbus1 pid 19 tid 100049 td 0xffffff000421a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735dcd30, rbp = 0 --- Tracing command usbus1 pid 18 tid 100048 td 0xffffff000421a390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735d7d30, rbp = 0 --- Tracing command usbus0 pid 17 tid 100047 td 0xffffff000421a720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735d2d30, rbp = 0 --- Tracing command usbus0 pid 16 tid 100046 td 0xffffff000421aab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735cdd30, rbp = 0 --- Tracing command usbus0 pid 15 tid 100045 td 0xffffff0002663ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735c8d30, rbp = 0 --- Tracing command usbus0 pid 14 tid 100044 td 0xffffff0003fe7000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x14b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735c3d30, rbp = 0 --- Tracing command sctp_iterator pid 8 tid 100043 td 0xffffff0003fe7390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 sctp_iterator_thread() at sctp_iterator_thread+0x54 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735bed30, rbp = 0 --- Tracing command fdc0 pid 7 tid 100040 td 0xffffff0003fe8000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735acd30, rbp = 0 --- Tracing command fw0_probe pid 6 tid 100038 td 0xffffff0003fe8720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 fw_bus_probe_thread() at fw_bus_probe_thread+0xbc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073594d30, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100025 td 0xffffff00024f4ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a3d30, rbp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xffffff00024f3000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800007bd30, rbp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xffffff00024f3720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000071d30, rbp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xffffff00024f3ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800006cd30, rbp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xffffff00024f4000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000067d30, rbp = 0 --- Tracing command intr pid 12 tid 100042 td 0xffffff0003fe7720 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xffffff0003fe7ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80735b1d30, rbp = 0 --- Tracing command intr pid 12 tid 100039 td 0xffffff0003fe8390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807359dd30, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xffffff0003fea000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073340d30, rbp = 0 --- Tracing command intr pid 12 tid 100035 td 0xffffff000265e720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807333bd30, rbp = 0 --- Tracing command intr pid 12 tid 100034 td 0xffffff000265eab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80732ced30, rbp = 0 --- Tracing command intr pid 12 tid 100033 td 0xffffff0002661000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80730bfd30, rbp = 0 --- Tracing command intr pid 12 tid 100032 td 0xffffff0002661390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8073076d30, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xffffff0002661720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff807302dd30, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xffffff0002661ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80729dcd30, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff0002663000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b7d30, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xffffff0002663390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b2d30, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff00024f4720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a8d30, rbp = 0 --- Tracing command intr pid 12 tid 100024 td 0xffffff000265c000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800009ed30, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff000265c390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000099d30, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xffffff000265cab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800008fd30, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00024f4390 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xffffff00024e4390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800005dd30, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xffffff00024e4720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000058d30, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xffffff00024e4ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000053d30, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xffffff00024ef000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800004ed30, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xffffff00024ef390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000049d30, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xffffff00024ef720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff808454a6, rsp = 0xffffff8000044b50, rbp = 0xffffff8000044b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff00024e3000 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff808454a6, rsp = 0xffffff8000017fe0, rbp = 0xffffff800003fb60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800003fd30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00024e3390 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff808454a6, rsp = 0xffffff800001efe0, rbp = 0xffffff800003ab60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800003ad30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff00024e3720 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x2c5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff808454a6, rsp = 0xffffff8000025fe0, rbp = 0xffffff8000035b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000035d30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00024e3ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x1d0 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c57c, rsp = 0x7fffffffe808, rbp = 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff00024e4000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800002bd30, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xffffff0003fe8ab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80734cfd30, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xffffff0002663720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000add30, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xffffff000265c720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000094d30, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xffffff000265e000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800008ad30, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xffffff000265e390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000085d30, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xffffff00024efab0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x5c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000080d30, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00024f3390 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000076d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c20d00 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x292 mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> call doadump Physical memory: 3951 MB Dumping 1509 MB: 1494 1478 1462 1446 1430 1414 1398 1382 1366 1350 1334 1318 1302 1286 1270 1254 1238 1222 1206 1190 1174 1158 1142 1126 1110 1094 1078 1062 1046 1030 1014 998 982 966 950 934 918 902 886 870 854 838 822 806 790 774 758 742 726 710 694 678 662 646 630 614 598 582 566 550 534 518 502 486 470 454 438 422 406 390 374 358 342 326 310 294 278 262 246 230 214 198 182 166 150 134 118 102 86 70 54 38 22 6 Dump complete = 0 db> reset (kgdb) bt #0 doadump () at pcpu.h:223 #1 0xffffffff801d930c in db_fncall (dummy1=Variable "dummy1" is not available. ) at ../../../ddb/db_command.c:548 #2 0xffffffff801d9641 in db_command (last_cmdp=0xffffffff80bee8a0, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #3 0xffffffff801d9890 in db_command_loop () at ../../../ddb/db_command.c:498 #4 0xffffffff801db869 in db_trap (type=Variable "type" is not available. ) at ../../../ddb/db_main.c:229 #5 0xffffffff805b40b5 in kdb_trap (type=0x3, code=0x0, tf=0xffffff8075c4a390) at ../../../kern/subr_kdb.c:534 #6 0xffffffff80867d61 in trap (frame=0xffffff8075c4a390) at ../../../amd64/amd64/trap.c:613 #7 0xffffffff8084da23 in calltrap () at ../../../amd64/amd64/exception.S:224 #8 0xffffffff805b428d in kdb_enter (why=0xffffffff80961033 "panic", msg=0xa
) at cpufunc.h:63 #9 0xffffffff805845ab in panic (fmt=Variable "fmt" is not available. ) at ../../../kern/kern_shutdown.c:558 #10 0xffffffff807d8026 in vm_page_free_toq (m=0xffffff00d50e28c0) at ../../../vm/vm_page.c:1379 #11 0xffffffff807d49a7 in vm_object_terminate (object=0xffffff00be2f4360) at ../../../vm/vm_object.c:726 #12 0xffffffff807d606a in vm_object_deallocate (object=0xffffff00be2f4360) at ../../../vm/vm_object.c:624 #13 0xffffffff807cdd30 in _vm_map_unlock (map=0xffffff00595747a8, file=Variable "file" is not available. ) at ../../../vm/vm_map.c:480 #14 0xffffffff807ce35f in vm_map_remove (map=0xffffff00595747a8, start=Variable "start" is not available. ) at ../../../vm/vm_map.c:2769 #15 0xffffffff807cf026 in vmspace_free (vm=0xffffff00595747a8) at ../../../vm/vm_map.c:329 #16 0xffffffff807d0cc6 in vmspace_exec (p=0xffffff0004eac8c0, minuser=Variable "minuser" is not available. ) at ../../../vm/vm_map.c:3450 #17 0xffffffff80558263 in exec_new_vmspace (imgp=0xffffff8075c4a9b0, sv=0xffffffff80bbb900) at ../../../kern/kern_exec.c:1022 #18 0xffffffff805405d6 in exec_elf64_imgact (imgp=0xffffff8075c4a9b0) at ../../../kern/imgact_elf.c:694 #19 0xffffffff80558742 in kern_execve (td=0xffffff0004e70720, args=0xffffff8075c4aae0, mac_p=Variable "mac_p" is not available. ) at ../../../kern/kern_exec.c:471 #20 0xffffffff808a8cad in freebsd32_execve (td=0xffffff0004e70720, uap=Variable "uap" is not available. ) at ../../../compat/freebsd32/freebsd32_misc.c:352 #21 0xffffffff808a462c in ia32_syscall (frame=0xffffff8075c4ac80) at ../../../amd64/ia32/ia32_syscall.c:182 #22 0xffffffff8084dfa5 in Xint0x80_syscall () at ia32_exception.S:72 #23 0x00000000280fe1fb in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 10 #10 0xffffffff807d8026 in vm_page_free_toq (m=0xffffff00d50e28c0) at ../../../vm/vm_page.c:1379 1379 KASSERT(!pmap_page_is_mapped(m), (kgdb) l 1374 vm_page_free_toq(vm_page_t m) 1375 { 1376 1377 if (VM_PAGE_GETQUEUE(m) != PQ_NONE) 1378 mtx_assert(&vm_page_queue_mtx, MA_OWNED); 1379 KASSERT(!pmap_page_is_mapped(m), 1380 ("vm_page_free_toq: freeing mapped page %p", m)); 1381 PCPU_INC(cnt.v_tfree); 1382 1383 if (m->busy || VM_PAGE_IS_FREE(m)) { (kgdb) p *m $1 = {pageq = {tqe_next = 0xffffff00d50e11c8, tqe_prev = 0xffffff00d2fc45a8}, listq = {tqe_next = 0x0, tqe_prev = 0xffffff00be2f43a8}, left = 0x0, right = 0x0, object = 0xffffff00be2f4360, pindex = 0x1d, phys_addr = 0xbc969000, md = {pv_list = { tqh_first = 0xffffff00bd078df0, tqh_last = 0xffffff00bd078df8}, pat_mode = 0x6}, queue = 0x2, segind = 0x1, flags = 0x90, order = 0xd, pool = 0x0, cow = 0x0, wire_count = 0x0, hold_count = 0x0, oflags = 0x0, act_count = 0x5, busy = 0x0, valid = 0xff, dirty = 0xff} (kgdb) $ svn diff /usr/src/sys -- Test scenario i386 stress2 with run.sh -a