GENERIC HEAD from Jan 11 2008 + pmap.?, vmcore.0 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-CURRENT #0: Sun Jan 13 15:20:29 EST 2008 test@beaver1.sentex.ca:/usr/pho/sys/amd64/compile/PHO WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz (2208.95-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fd Stepping = 13 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xe39d<SSE3,RSVD2,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> Cores per package: 2 usable memory = 3206316032 (3057 MB) avail memory = 3104120832 (2960 MB) ACPI APIC Table: <IntelR AWRDACPI> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 4 : Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted Loading configuration files. kernel dumps on /dev/ad0s1b Entropy harvesting: interrupts ethernet point_to_point kickstart. swapon: adding /dev/ad0s1b as swap device Starting file system checks: /dev/ad0s1a: 2484 files, 218147 used, 35668 free (316 frags, 4419 blocks, 0.1% fragmentation) /dev/ad0s1e: DEFER FOR BACKGROUND CHECKING /dev/ad0s1f: DEFER FOR BACKGROUND CHECKING /dev/ad0s1d: DEFER FOR BACKGROUND CHECKING Setting hostuuid: 00e04d0e-037b-ffff-ffff-ffffffffffff. Setting hostid: WARNING: /tmp was not properly dismounted 0xc1d5bca1. Mounting local fWARNING: /usr was not properly dismounted ile systems:/usr: mount pending error: blocks 403896 files 27265 WARNING: /var was not properly dismounted /var: mount pending error: blocks 212 files 1 . Setting hostname: beaver1.sentex.ca. net.inet6.ip6.auto_linklocal: 1 -> 0 re0: link state changed to DOWN re0: no link .....re0: link state changed to UP got link re0: link state chDHCPREQUEST on rae0 to 255.255.25ng5.255 port 67ed to DOWN re0DHCPREQUEST on r:e0 to 255.255.25 5.255 port 67l iDHCPACK from 192nk.168.5.1 s tate changed to UP re0: link state changed to DOWN bound to 192.168.5.112 -- renewal in 300 seconds. lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:e0:4d:0e:03:7b inet 192.168.5.112 netmask 0xffffff00 broadcast 192.168.5.255 media: Ethernet autoselect (none) status: no carrier Additional routing options: IP gateway=YES. Starting devd. hw.acpi.cpu.cx_lowest: C1 sysctl: hw.acpi.cpu.cx_lowest: Invalid argument Additional IP options:. Mounting NFS file systems:. Creating and/or trimming log files:. Starting syslogd. Checking for core dump on /dev/ad0s1b... savecore: no dumps found ELF ldconfig path: /lib /usr/lib /usr/lib/compat 32-bit compatibility ldconfig path: Initial amd64 initialization:. Additional ABI support:. Clearing /tmp (X related). Starting local daemons:. Updating motd. Mounting late file systems:. Configuring syscons: blanktime. Starting sshd. Starting cron. Local package initialization:. Starting background file system checks in 60 seconds. Mon Jan 14 18:02:55 EST 2008 FreeBSD/amd64 (beaver1.sentex.ca) (ttyd0) Tue Jan 15 17:49:52 CET 2008 5:49PM up 17:47, 2 users, load averages: 0.09, 15.48, 43.85 Tue Jan 15 18:05:28 CET 2008 6:05PM up 18:03, 2 users, load averages: 0.01, 0.83, 15.43 Jan 15 12:15:45 beaver1 su: test to root on /dev/ttyp0 TPTE at 0xffff8000000028e0 IS ZERO @ VA 0051c000 panic: bad pte cpuid = 0 KDB: enter: panic [thread pid 47647 tid 100059 ] Stopped at kdb_enter+0x3d: movq $0,0x609258(%rip) db> bt Tracing pid 47647 tid 100059 td 0xffffff0001cd6000 kdb_enter() at kdb_enter+0x3d panic() at panic+0x176 pmap_remove_pages() at pmap_remove_pages+0x62b vmspace_exit() at vmspace_exit+0x97 exit1() at exit1+0x3a6 sigexit() at sigexit+0x9f postsig() at postsig+0x38e ast() at ast+0x357 Xfast_syscall() at Xfast_syscall+0xe0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8005380ec, rsp = 0x7fffffffe878, rbp = 0x7fffffffec18 --- db> show allpcpu Current CPU: 0 cpuid = 0 curthread = 0xffffff0001cd6000: pid 47647 "sh" curpcb = 0xffffffffaf1a8d40 fpcurthread = none idlethread = 0xffffff00010e8360: pid 11 "idle: cpu0" spin locks held: cpuid = 1 curthread = 0xffffff00010e86c0: pid 11 "idle: cpu1" curpcb = 0xffffffffac45cd40 fpcurthread = none idlethread = 0xffffff00010e86c0: pid 11 "idle: cpu1" spin locks held: db> show alllocks Process 47647 (sh) thread 0xffffff0001cd6000 (100059) exclusive sleep mutex pmap r = 0 (0xffffff002972e5f0) locked @ amd64/amd64/pmap.c:3449 exclusive sleep mutex vm page queue mutex r = 0 (0xffffffff80b50c60) locked @ amd64/amd64/pmap.c:3448 Process 47646 (initial thread) thread 0xffffff001d2b0360 (100169) exclusive sx user map r = 0 (0xffffff001aecdc08) locked @ vm/vm_mmap.c:574 Process 2930 (sshd) thread 0xffffff0001d076c0 (100065) exclusive sx so_rcv_sx r = 0 (0xffffff0001d26648) locked @ kern/uipc_sockbuf.c:145 Process 2726 (syslogd) thread 0xffffff0001dbd000 (100096) exclusive sleep mutex vm object (standard object) r = 0 (0xffffff001df945e8) locked @ kern/vfs_bio.c:3496 db> ps pid ppid pgrp uid state wmesg wchan cmd 47647 47546 47647 0 RE+ CPU 0 sh 47646 47639 47639 0 L+ *vm page 0xffffff00b6ac7aa0 initial thread 47639 47546 47639 0 S+ wait 0xffffff001d2ac000 sh 47638 47631 47631 0 R+ initial thread 47631 47546 47631 0 S+ wait 0xffffff001af38468 sh 47630 47623 47623 0 R+ initial thread 47623 47546 47623 0 S+ wait 0xffffff0001dcb000 sh 47546 47545 47546 0 S+ ttyin 0xffffff0001292410 csh 47545 2934 47545 1002 S+ wait 0xffffff00059128d0 su 47407 2926 2926 1002 S+ nanslp 0xffffffff80aae5c8 sleep 2946 1 2946 0 Ss nanslp 0xffffffff80aae5c8 watchdogd 2934 2933 2934 1002 Ss+ pause 0xffffff0001ffe990 csh 2933 2930 2930 1002 R sshd 2930 2845 2930 0 Ss sbwait 0xffffff0001d266a4 sshd 2926 2923 2926 1002 S+ wait 0xffffff0001ead468 sh 2923 2919 2923 1002 S+ pause 0xffffff00150020c0 csh 2919 1 2919 0 Ss+ wait 0xffffff0001eaf8d0 login 2918 1 2918 0 Ss+ ttyin 0xffffff0001b44810 getty 2917 1 2917 0 Ss+ ttyin 0xffffff0001b44c10 getty --More-- 2916 1 2916 0 Ss+ ttyin 0xffffff0001b5a010 getty 2915 1 2915 0 Ss+ ttyin 0xffffff0001b5a410 getty 2914 1 2914 0 Ss+ ttyin 0xffffff0001b5a810 getty 2913 1 2913 0 Ss+ ttyin 0xffffff0001b5ac10 getty 2912 1 2912 0 Ss+ ttyin 0xffffff0001b5b010 getty 2911 1 2911 0 Ss+ ttyin 0xffffff0001b5b410 getty 2871 1 2871 0 Ss nanslp 0xffffffff80aae5c8 cron 2865 1 2865 25 Ss pause 0xffffff0001dcd0c0 sendmail 2861 1 2861 0 Ss select 0xffffff0001be90d0 sendmail 2845 1 2845 0 Ss select 0xffffff0001b591d0 sshd 2726 1 2726 0 Ls *vm page 0xffffff00b6ac7aa0 syslogd 2681 1 2681 0 Ss select 0xffffff0001e70550 devd 2615 1 2615 65 Ss select 0xffffff0001f803d0 dhclient 2569 1 2348 0 S+ select 0xffffff0001f7cad0 dhclient 2448 1 2448 0 Ss pause 0xffffff0001eae0c0 adjkerntz 26 0 0 0 SL sdflush 0xffffffff80b4ff28 [softdepflush] 25 0 0 0 SL syncer 0xffffffff80aae240 [syncer] 24 0 0 0 SL vlruwt 0xffffff0001bb2468 [vnlru] 23 0 0 0 SL psleep 0xffffffff80b4059c [bufdaemon] 22 0 0 0 SL pgzero 0xffffffff80b519cc [pagezero] --More-- 21 0 0 0 SL psleep 0xffffffff80b50ce8 [vmdaemon] 20 0 0 0 SL psleep 0xffffffff80b50cac [pagedaemon] 19 0 0 0 SL waiting_ 0xffffffff80b43ca8 [sctp_iterator] 18 0 0 0 SL tzpoll 0xffffffff80a7c1b0 [acpi_thermal] 17 0 0 0 SL usbtsk 0xffffffff80aa9ac8 [usbtask-dr] 16 0 0 0 SL usbtsk 0xffffffff80aa9aa0 [usbtask-hc] 15 0 0 0 SL usbevt 0xffffffff80e54420 [usb0] 14 0 0 0 SL - 0xffffff00011fb880 [acpi_task_2] 9 0 0 0 SL - 0xffffff00011fb880 [acpi_task_1] 8 0 0 0 SL - 0xffffff00011fb880 [acpi_task_0] 7 0 0 0 SL - 0xffffff00011fb900 [kqueue taskq] 6 0 0 0 SL ccb_scan 0xffffffff80a77c20 [xpt_thrd] 5 0 0 0 SL - 0xffffff00011fbd80 [thread taskq] 13 0 0 0 SL - 0xffffffff80aae248 [yarrow] 4 0 0 0 SL - 0xffffffff80aaab78 [g_down] 3 0 0 0 SL - 0xffffffff80aaab70 [g_up] 2 0 0 0 SL - 0xffffffff80aaab60 [g_event] 12 0 0 0 WL (threaded) intr 100032 I [irq1: atkbd0] 100031 I [swi0: sio] --More-- 100029 I [irq19: atapci1] 100028 I [irq15: ata1] 100027 I [irq14: ata0] 100023 I [irq23: uhci0] 100022 I [irq9: acpi0] 100017 I [swi2: cambio] 100015 I [swi6: task queue] 100014 I [swi6: Giant taskq] 100012 I [swi5: +] 100007 I [swi3: vm] 100006 I [swi4: clock sio] 100005 I [swi1: net] 11 0 0 0 RL (threaded) idle 100004 CanRun [idle: cpu0] 100003 Run CPU 1 [idle: cpu1] 1 0 1 0 SLs wait 0xffffff00010e58d0 [init] 10 0 0 0 SL audit_wo 0xffffffff80b4f400 [audit] 0 0 0 0 WLs [swapper] db> set $lines 0 db> allt Tracing command sh pid 47647 tid 100059 td 0xffffff0001cd6000 kdb_enter() at kdb_enter+0x3d panic() at panic+0x176 pmap_remove_pages() at pmap_remove_pages+0x62b vmspace_exit() at vmspace_exit+0x97 exit1() at exit1+0x3a6 sigexit() at sigexit+0x9f postsig() at postsig+0x38e ast() at ast+0x357 Xfast_syscall() at Xfast_syscall+0xe0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x8005380ec, rsp = 0x7fffffffe878, rbp = 0x7fffffffec18 --- Tracing command tlbtest pid 47646 tid 100169 td 0xffffff001d2b0360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 turnstile_wait() at turnstile_wait+0x24d _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xe1 pmap_remove() at pmap_remove+0x53 vm_map_delete() at vm_map_delete+0xdf munmap() at munmap+0xc1 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (73, FreeBSD ELF64, munmap), rip = 0x8008116cc, rsp = 0x7fffffffeb68, rbp = 0x7fffffffebb0 --- Tracing command sh pid 47639 tid 100164 td 0xffffff001d2b1360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092b51c, rsp = 0x7fffffffe858, rbp = 0xba17 --- Tracing command tlbtest pid 47638 tid 100181 td 0xffffff001d258360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80080bbdc, rsp = 0x7fffffffeaf8, rbp = 0x7fffffffeb60 --- Tracing command sh pid 47631 tid 100124 td 0xffffff0001dbda20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092b51c, rsp = 0x7fffffffe858, rbp = 0xba0f --- Tracing command tlbtest pid 47630 tid 100288 td 0xffffff00b77b8000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ipi_bitmap_handler() at ipi_bitmap_handler+0x8a Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76 --- interrupt, rip = 0xffffffff80742e9b, rsp = 0xffffffffaf61cc60, rbp = 0x7fffffffeb60 --- Xfast_syscall() at Xfast_syscall+0xdb Tracing command sh pid 47623 tid 100072 td 0xffffff0001dfb000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092b51c, rsp = 0x7fffffffe858, rbp = 0xba07 --- Tracing command csh pid 47546 tid 100167 td 0xffffff001d2b06c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 ttysleep() at ttysleep+0x25 ttread() at ttread+0x303 ptsread() at ptsread+0x37 giant_read() at giant_read+0x51 devfs_read_f() at devfs_read_f+0x75 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x54 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (3, FreeBSD ELF64, read), rip = 0x8009cfdec, rsp = 0x7fffffffe618, rbp = 0x1 --- Tracing command su pid 47545 tid 103658 td 0xffffff008fbb5360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009c851c, rsp = 0x7fffffffe688, rbp = 0xb9ba --- Tracing command sleep pid 47407 tid 100318 td 0xffffff008fde5000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8006f4bdc, rsp = 0x7fffffffec08, rbp = 0x7fffffffec90 --- Tracing command watchdogd pid 2946 tid 100055 td 0xffffff0001baf360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80091dbdc, rsp = 0x7fffffffea68, rbp = 0x7fffffffeac0 --- Tracing command csh pid 2934 tid 100088 td 0xffffff0001dbea20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094395c, rsp = 0x7fffffffe648, rbp = 0x800c6e040 --- Tracing command sshd pid 2933 tid 100054 td 0xffffff0001baf6c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x8016f5d6c, rsp = 0x7fffffffe518, rbp = 0x7fffffffe590 --- Tracing command sshd pid 2930 tid 100065 td 0xffffff0001d076c0 Tracing command sh pid 2926 tid 100082 td 0xffffff0001e87360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80092b51c, rsp = 0x7fffffffe7b8, rbp = 0xb6e --- Tracing command csh pid 2923 tid 100087 td 0xffffff0001e65000 Tracing command login pid 2919 tid 100075 td 0xffffff0001df9360 Tracing command getty pid 2918 tid 100074 td 0xffffff0001df96c0 Tracing command getty pid 2917 tid 100051 td 0xffffff0001bac6c0 Tracing command getty pid 2916 tid 100052 td 0xffffff0001bac360 Tracing command getty pid 2915 tid 100043 td 0xffffff0001c3a360 Tracing command getty pid 2914 tid 100090 td 0xffffff0001dbe360 Tracing command getty pid 2913 tid 100097 td 0xffffff0001d1ca20 Tracing command getty pid 2912 tid 100056 td 0xffffff0001cd6a20 Tracing command getty pid 2911 tid 100068 td 0xffffff0001dfc000 Tracing command cron pid 2871 tid 100060 td 0xffffff0001cafa20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24c kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800912bdc, rsp = 0x7fffffffec28, rbp = 0x3a --- Tracing command sendmail pid 2865 tid 100066 td 0xffffff0001d07360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d1095c, rsp = 0x7fffffffda38, rbp = 0x1 --- Tracing command sendmail pid 2861 tid 100070 td 0xffffff0001dfb6c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x800d9cd6c, rsp = 0x7fffffffcf58, rbp = 0x7fffffffd7f0 --- Tracing command sshd pid 2845 tid 100083 td 0xffffff0001e87000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (93, FreeBSD ELF64, select), rip = 0x8016f5d6c, rsp = 0x7fffffffe618, rbp = 0x2 --- Tracing command syslogd pid 2726 tid 100096 td 0xffffff0001dbd000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 turnstile_wait() at turnstile_wait+0x24d _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xe1 bdwrite() at bdwrite+0x1f5 ffs_write() at ffs_write+0x38e VOP_WRITE_APV() at VOP_WRITE_APV+0x131 vn_write() at vn_write+0x244 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 writev() at writev+0x41 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (121, FreeBSD ELF64, writev), rip = 0x8008106ac, rsp = 0x7fffffffcc88, rbp = 0 --- Tracing command devd pid 2681 tid 100046 td 0xffffff0001c376c0 Tracing command dhclient pid 2615 tid 100045 td 0xffffff0001c37a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x16e seltdwait() at seltdwait+0x56 poll() at poll+0x432 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d075c, rsp = 0x7fffffffed98, rbp = 0x493e0 --- Tracing command dhclient pid 2569 tid 100049 td 0xffffff0001c37000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x15f seltdwait() at seltdwait+0xac poll() at poll+0x432 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006d075c, rsp = 0x7fffffffed98, rbp = 0x5 --- Tracing command adjkerntz pid 2448 tid 100080 td 0xffffff0001df8000 Tracing command softdepflush pid 26 tid 100040 td 0xffffff0001bad000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e softdep_flush() at softdep_flush+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface46d30, rbp = 0 --- Tracing command syncer pid 25 tid 100039 td 0xffffff0001bad360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sched_sync() at sched_sync+0x61a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface41d30, rbp = 0 --- Tracing command vnlru pid 24 tid 100038 td 0xffffff0001bad6c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vnlru_proc() at vnlru_proc+0x69f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface3cd30, rbp = 0 --- Tracing command bufdaemon pid 23 tid 100037 td 0xffffff0001bada20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e buf_daemon() at buf_daemon+0x208 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface37d30, rbp = 0 --- Tracing command pagezero pid 22 tid 100036 td 0xffffff0001baf000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface32d30, rbp = 0 --- Tracing command vmdaemon pid 21 tid 100035 td 0xffffff00012256c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface2dd30, rbp = 0 --- Tracing command pagedaemon pid 20 tid 100034 td 0xffffff0001225a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e vm_pageout() at vm_pageout+0xadd fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface28d30, rbp = 0 --- Tracing command sctp_iterator pid 19 tid 100033 td 0xffffff0001b00000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 sctp_iterator_thread() at sctp_iterator_thread+0x4a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffface23d30, rbp = 0 --- Tracing command acpi_thermal pid 18 tid 100030 td 0xffffff0001b00a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e acpi_tz_thread() at acpi_tz_thread+0x1dd fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffacdf1d30, rbp = 0 --- Tracing command usbtask-dr pid 17 tid 100026 td 0xffffff000121f6c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac591d30, rbp = 0 --- Tracing command usbtask-hc pid 16 tid 100025 td 0xffffff000121fa20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 usb_task_thread() at usb_task_thread+0x94 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac58cd30, rbp = 0 --- Tracing command usb0 pid 15 tid 100024 td 0xffffff0001224000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e usb_event_thread() at usb_event_thread+0xaa fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac587d30, rbp = 0 --- Tracing command acpi_task_2 pid 14 tid 100021 td 0xffffff0001224a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4b7d30, rbp = 0 --- Tracing command acpi_task_1 pid 9 tid 100020 td 0xffffff0001225000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4b2d30, rbp = 0 --- Tracing command acpi_task_0 pid 8 tid 100019 td 0xffffff0001225360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4add30, rbp = 0 --- Tracing command kqueue taskq pid 7 tid 100018 td 0xffffff00010f4a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac4a8d30, rbp = 0 --- Tracing command xpt_thrd pid 6 tid 100016 td 0xffffff000121e360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac49ed30, rbp = 0 --- Tracing command thread taskq pid 5 tid 100013 td 0xffffff000121f000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _sleep() at _sleep+0x351 taskqueue_thread_loop() at taskqueue_thread_loop+0x71 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac48fd30, rbp = 0 --- Tracing command yarrow pid 13 tid 100011 td 0xffffff00010e9360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e random_kthread() at random_kthread+0x194 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac485d30, rbp = 0 --- Tracing command g_down pid 4 tid 100010 td 0xffffff00010e96c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac480d30, rbp = 0 --- Tracing command g_up pid 3 tid 100009 td 0xffffff00010e9a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac47bd30, rbp = 0 --- Tracing command g_event pid 2 tid 100008 td 0xffffff00010f4000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_timedwait() at sleepq_timedwait+0x3c _sleep() at _sleep+0x33e g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac476d30, rbp = 0 --- Tracing command intr pid 12 tid 100032 td 0xffffff0001b00360 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xffffff0001b006c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffacdf6d30, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff0001b01000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xffffff0001b01360 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100027 td 0xffffff0001b016c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffacda9d30, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff0001224360 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100022 td 0xffffff00012246c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100017 td 0xffffff000121e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100015 td 0xffffff000121e6c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac499d30, rbp = 0 --- Tracing command intr pid 12 tid 100014 td 0xffffff000121ea20 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xffffff000121f360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac48ad30, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xffffff00010f4360 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100006 td 0xffffff00010f46c0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac46cd30, rbp = 0 --- Tracing command intr pid 12 tid 100005 td 0xffffff00010e8000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 ithread_loop() at ithread_loop+0x390 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac467d30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00010e8360 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sched_idletd() at sched_idletd+0x19f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac461d30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff00010e86c0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x378 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff8074bd0a, rsp = 0xffffffffab99fff0, rbp = 0xffffffffac45cbe0 --- spinlock_enter() at spinlock_enter+0x1a sched_idletd() at sched_idletd+0x70 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac45cd30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00010e8a20 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_catch_signals() at sleepq_catch_signals+0x25a sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40bf4c, rsp = 0x7fffffffe8e8, rbp = 0x7fffffffed70 --- Tracing command audit pid 10 tid 100001 td 0xffffff00010e9000 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x15e audit_worker() at audit_worker+0x86 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffac452d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80aab0f0 sched_switch() at sched_switch+0x2a1 mi_switch() at mi_switch+0x233 scheduler() at scheduler+0x3c1 mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> call doadump Physical memory: 3057 MB Dumping 328 MB: 313 297 281 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9 Dump complete = 0xf db> c Uptime: 18h15m47s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... cpu_reset: Stopping other CPUs (kgdb) l *pmap_remove_pages+0x62b 0xffffffff8075318b is in pmap_remove_pages (../../../amd64/amd64/pmap.c:3491). 3486 KASSERT(m->phys_addr == (tpte & PG_FRAME), 3487 ("vm_page_t %p phys_addr mismatch %016jx %016jx", 3488 m, (uintmax_t)m->phys_addr, 3489 (uintmax_t)tpte)); 3490 3491 KASSERT(m < &vm_page_array[vm_page_array_size], 3492 ("pmap_remove_pages: bad tpte %#jx", 3493 (uintmax_t)tpte)); 3494 3495 pte_clear(pte); -- Test scenario: TLB test program from ups@