Git branch amd64_ctx 5fb9257, vmcore.2 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-CURRENT #1: Sat Nov 1 09:17:06 CET 2008 pho@x4.osted.lan:/var/tmp/deviant2/sys/amd64/compile/PHO WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1799.99-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff,,,Prefetch,,> TSC: P-state invariant Cores per package: 4 usable memory = 4145442816 (3953 MB) avail memory = 3986247680 (3801 MB) : Trying to mount root from ufs:/dev/ad4s2a lock order reversal: 1st 0xffffff00014d8070 user map (user map) @ vm/vm_map.c:3115 2nd 0xffffff0004d48578 ufs (ufs) @ kern/vfs_subr.c:2046 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd1b ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x92 vnode_pager_lock() at vnode_pager_lock+0x1d3 vm_fault() at vm_fault+0x1e2 trap_pfault() at trap_pfault+0x128 trap() at trap+0x51c calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x40014f, rsp = 0x7fffffffee50, rbp = 0x7fffffffee70 --- Loading configuration files. kernel dumps on /dev/ad4s2b Entropy harvesting: interrupts ethernet point_to_point kickstart. swapon: adding /dev/ad4s1b as swap device swapon: adding /dev/ad4s2b as swap device Starting file system checks: /dev/ad4s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2a: clean, 687080 free (1472 frags, 85701 blocks, 0.1% fragmentation) /dev/ad4s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2e: clean, 50775849 free (665 frags, 6346898 blocks, 0.0% fragmentation) /dev/ad4s2f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2f: clean, 1538647 free (45279 frags, 186671 blocks, 0.9% fragmentation) /dev/ad4s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4s2d: clean, 49179054 free (10278 frags, 6146097 blocks, 0.0% fragmentation) Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Mounting local file systems:. Setting hostname: x4.osted.lan. net.inet6.ip6.auto_linklocal: 1 -> 0 re0: link state changed to DOWN lo0: flags=8049 metric 0 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 re0: flags=8843 metric 0 mtu 1500 options=389b ether 00:21:85:15:33:7d inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier add net default: gateway 192.168.1.1 Additional routing options:. Starting devd. hw.acpi.cpu.cx_lowest: C1 -> C1 Additional IP options:. Mounting NFS file systems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files:. Starting syslogd. Checking for core dump on /dev/ad4s2b... savecore: no dumps found Initial amd64 initialization:. Additional ABI support:. Starting rpcbind. Clearing /tmp (X related). Starting mountd. Starting nfsd. Starting local daemons:. Updating motd. Mounting late file systems:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization:. Sat Nov 1 09:26:06 CET 2008 Nov 1 09:29:13 x4 su: pho to root on /dev/pts/0 Nov 1 11:46:02 x4 shutdown: power-down by pho: Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex cdev (cdev) r = 0 (0xffffffff80b55780) locked @ kern/kern_conf.c:71 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c2 trap() at trap+0x2a3 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff804b69a3, rsp = 0xfffffffeb668f570, rbp = 0xfffffffeb668f590 --- dev_refthread() at dev_refthread+0x33 cn_devopen() at cn_devopen+0x4e cnopen() at cnopen+0x42 giant_open() at giant_open+0x6b devfs_open() at devfs_open+0x184 VOP_OPEN_APV() at VOP_OPEN_APV+0xaf vn_open_cred() at vn_open_cred+0x308 kern_openat() at kern_openat+0x15e syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (5, FreeBSD ELF64, open), rip = 0x447d0c, rsp = 0x7fffffffe728, rbp = 0x7fffffffe7b0 --- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0xb8 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff804b69a3 stack pointer = 0x10:0xfffffffeb668f570 frame pointer = 0x10:0xfffffffeb668f590 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1111 (init) [thread pid 1111 tid 100117 ] Stopped at dev_refthread+0x33: movq 0xb8(%r12),%rbx db> set $lines 0 db> bt Tracing pid 1111 tid 100117 td 0xffffff0004ec7a50 dev_refthread() at dev_refthread+0x33 cn_devopen() at cn_devopen+0x4e cnopen() at cnopen+0x42 giant_open() at giant_open+0x6b devfs_open() at devfs_open+0x184 VOP_OPEN_APV() at VOP_OPEN_APV+0xaf vn_open_cred() at vn_open_cred+0x308 kern_openat() at kern_openat+0x15e syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (5, FreeBSD ELF64, open), rip = 0x447d0c, rsp = 0x7fffffffe728, rbp = 0x7fffffffe7b0 --- db> show allpcpu Current CPU: 2 cpuid = 0 curthread = 0xffffff0004eaca50: pid 805 "getty" curpcb = 0xfffffffeb6595d40 fpcurthread = none idlethread = 0xffffff00014de6e0: pid 11 "idle: cpu0" curpmap = 0 tssp = 0xffffffff80d33b80 commontssp = 0xffffffff80d33b80 rsp0 = 0xfffffffeb6595d40 gs32p = 0xffffffff80d32ae0 ldt = 0xffffffff80d32af0 tss = 0xffffffff80d32ad0 spin locks held: cpuid = 1 curthread = 0xffffff00014d4a50: pid 12 "swi4: clock" curpcb = 0xfffffffe40044d40 fpcurthread = none idlethread = 0xffffff00014d3000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff80d33be8 commontssp = 0xffffffff80d33be8 rsp0 = 0xfffffffe40044d40 gs32p = 0xffffffff80d32b40 ldt = 0xffffffff80d32b50 tss = 0xffffffff80d32b30 spin locks held: cpuid = 2 curthread = 0xffffff0004ec7a50: pid 1111 "init" curpcb = 0xfffffffeb668fd40 fpcurthread = none idlethread = 0xffffff00014d3370: pid 11 "idle: cpu2" curpmap = 0 tssp = 0xffffffff80d33c50 commontssp = 0xffffffff80d33c50 rsp0 = 0xfffffffeb668fd40 gs32p = 0xffffffff80d32ba0 ldt = 0xffffffff80d32bb0 tss = 0xffffffff80d32b90 spin locks held: cpuid = 3 curthread = 0xffffff0004e22a50: pid 800 "getty" curpcb = 0xfffffffeb6590d40 fpcurthread = none idlethread = 0xffffff00014d36e0: pid 11 "idle: cpu3" curpmap = 0 tssp = 0xffffffff80d33cb8 commontssp = 0xffffffff80d33cb8 rsp0 = 0xfffffffeb6590d40 gs32p = 0xffffffff80d32c00 ldt = 0xffffffff80d32c10 tss = 0xffffffff80d32bf0 spin locks held: db> show alllocks Process 1111 (init) thread 0xffffff0004ec7a50 (100117) exclusive sleep mutex cdev (cdev) r = 0 (0xffffffff80b55780) locked @ kern/kern_conf.c:71 exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff80b57a60) locked @ dev/kbdmux/kbdmux.c:1044 Process 808 (sshd) thread 0xffffff0004eab370 (100105) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0004ec8d68) locked @ kern/uipc_sockbuf.c:148 Process 807 (getty) thread 0xffffff0004fb7000 (100116) exclusive lockmgr devfs (devfs) r = 0 (0xffffff00141b9098) locked @ kern/vfs_vnops.c:293 Process 806 (getty) thread 0xffffff0004fb7370 (100115) exclusive lockmgr devfs (devfs) r = 0 (0xffffff00141b9308) locked @ kern/vfs_vnops.c:293 Process 805 (getty) thread 0xffffff0004eaca50 (100067) exclusive sleep mutex vnode interlock (vnode interlock) r = 0 (0xffffff00141b9638) locked @ fs/devfs/devfs_vnops.c:419 exclusive sleep mutex session (session) r = 0 (0xffffff0004dd8a40) locked @ fs/devfs/devfs_vnops.c:418 exclusive sx proctree (proctree) r = 0 (0xffffffff80b57ba0) locked @ fs/devfs/devfs_vnops.c:416 exclusive lockmgr devfs (devfs) r = 0 (0xffffff00141b9578) locked @ kern/vfs_vnops.c:293 Process 800 (getty) thread 0xffffff0004e22a50 (100066) exclusive sleep mutex devfs interlock (devfs interlock) r = 0 (0xffffffff80b53e80) locked @ fs/devfs/devfs_vnops.c:1164 Process 12 (intr) thread 0xffffff00014e36e0 (100026) exclusive sleep mutex re0 (network driver) r = 0 (0xfffffffe403ce9c8) locked @ dev/re/if_re.c:2115 db> show lockedvnods Locked vnodes 0xffffff00141b94e0: tag devfs, type VCHR usecount 2, writecount 0, refcount 2 mountedhere 0xffffff0004d42600 flags (VV_ISTTY) lock type devfs: EXCL by thread 0xffffff0004eaca50 (pid 805) #0 0xffffffff804d73f8 at __lockmgr_args+0x758 #1 0xffffffff805650e9 at vop_stdlock+0x39 #2 0xffffffff8080995b at VOP_LOCK1_APV+0x9b #3 0xffffffff80580ad7 at _vn_lock+0x57 #4 0xffffffff80581be1 at vn_close+0x71 #5 0xffffffff80581d20 at vn_closefile+0x80 #6 0xffffffff8047a7d8 at devfs_close_f+0x28 #7 0xffffffff804be4f3 at _fdrop+0x23 #8 0xffffffff804bfb4d at closef+0x4d #9 0xffffffff804c0468 at fdfree+0x178 #10 0xffffffff804ca165 at exit1+0x305 #11 0xffffffff804ec4cf at sigexit+0x8f #12 0xffffffff804ed0ef at postsig+0x32f #13 0xffffffff80525167 at ast+0x367 #14 0xffffffff807a6739 at doreti_ast+0x1f dev ttyv5 0xffffff00141b9270: tag devfs, type VCHR usecount 2, writecount 0, refcount 2 mountedhere 0xffffff0004d42400 flags (VV_ISTTY) lock type devfs: EXCL by thread 0xffffff0004fb7370 (pid 806) #0 0xffffffff804d73f8 at __lockmgr_args+0x758 #1 0xffffffff805650e9 at vop_stdlock+0x39 #2 0xffffffff8080995b at VOP_LOCK1_APV+0x9b #3 0xffffffff80580ad7 at _vn_lock+0x57 #4 0xffffffff80581be1 at vn_close+0x71 #5 0xffffffff80581d20 at vn_closefile+0x80 #6 0xffffffff8047a7d8 at devfs_close_f+0x28 #7 0xffffffff804be4f3 at _fdrop+0x23 #8 0xffffffff804bfb4d at closef+0x4d #9 0xffffffff804c0468 at fdfree+0x178 #10 0xffffffff804ca165 at exit1+0x305 #11 0xffffffff804ec4cf at sigexit+0x8f #12 0xffffffff804ed0ef at postsig+0x32f #13 0xffffffff80525167 at ast+0x367 #14 0xffffffff807a6739 at doreti_ast+0x1f dev ttyv6 0xffffff00141b9000: tag devfs, type VCHR usecount 2, writecount 0, refcount 2 mountedhere 0xffffff0004d42200 flags (VV_ISTTY) lock type devfs: EXCL by thread 0xffffff0004fb7000 (pid 807) #0 0xffffffff804d73f8 at __lockmgr_args+0x758 #1 0xffffffff805650e9 at vop_stdlock+0x39 #2 0xffffffff8080995b at VOP_LOCK1_APV+0x9b #3 0xffffffff80580ad7 at _vn_lock+0x57 #4 0xffffffff80581be1 at vn_close+0x71 #5 0xffffffff80581d20 at vn_closefile+0x80 #6 0xffffffff8047a7d8 at devfs_close_f+0x28 #7 0xffffffff804be4f3 at _fdrop+0x23 #8 0xffffffff804bfb4d at closef+0x4d #9 0xffffffff804c0468 at fdfree+0x178 #10 0xffffffff804ca165 at exit1+0x305 #11 0xffffffff804ec4cf at sigexit+0x8f #12 0xffffffff804ed0ef at postsig+0x32f #13 0xffffffff80525167 at ast+0x367 #14 0xffffffff807a6739 at doreti_ast+0x1f dev ttyv7 db> ps pid ppid pgrp uid state wmesg wchan cmd 1111 1 1 0 R CPU 2 init 819 818 819 0 S+ tty inpu 0xffffff0004d29cb8 csh 818 811 818 1001 S+ wait 0xffffff0004eca000 su 811 810 811 1001 Ss+ wait 0xffffff00141bb448 bash 810 808 808 1001 S select 0xffffff001421c0c0 sshd 808 749 808 0 Ss sbwait 0xffffff0004ec8db4 sshd 807 1 807 0 REs+ getty 806 1 806 0 REs+ getty 805 1 805 0 REs+ CPU 0 getty 804 1 804 0 Rs+ getty 803 1 803 0 LEs+ *Giant 0xffffff0004d2e5a0 getty 802 1 802 0 LEs+ *Giant 0xffffff0004d2e5a0 getty 800 1 800 0 REs+ CPU 3 getty 765 1 765 0 Ss nanslp 0xffffffff80b58528 cron 759 1 759 25 Ss pause 0xffffff00141bf4e8 sendmail 755 1 755 0 Ss select 0xffffff0004d98440 sendmail 749 1 749 0 Ss select 0xffffff0004dd72c0 sshd 728 1 728 0 Ss select 0xffffff0004ee5640 ntpd 660 656 656 0 S - 0xffffff0004f95200 nfsd 659 656 656 0 S - 0xffffff0004ed5a00 nfsd 658 656 656 0 S - 0xffffff0004eeb600 nfsd 657 656 656 0 S - 0xffffff0004ed4400 nfsd 656 1 656 0 Ss select 0xffffff0004f03cc0 nfsd 654 1 654 0 Ss select 0xffffff0004dd7040 mountd 603 1 603 0 Ss select 0xffffff0004d4b740 rpcbind 587 1 587 0 Ss select 0xffffff0004d99940 syslogd 532 1 532 0 Ss select 0xffffff0004da52c0 devd 27 0 0 0 SL sdflush 0xffffffff80d292f8 [softdepflush] 26 0 0 0 SL syncer 0xffffffff80d19820 [syncer] 25 0 0 0 SL vlruwt 0xffffff0004daa000 [vnlru] 24 0 0 0 SL psleep 0xffffffff80d1935c [bufdaemon] 23 0 0 0 SL pgzero 0xffffffff80d2ac6c [pagezero] 22 0 0 0 SL psleep 0xffffffff80d2a008 [vmdaemon] 9 0 0 0 SL psleep 0xffffffff80d29fcc [pagedaemon] 8 0 0 0 SL waiting_ 0xffffffff80d1c6e8 [sctp_iterator] 7 0 0 0 SL - 0xffffff0004d2a848 [fdc0] 6 0 0 0 SL - 0xfffffffe403db000 [fw0_probe] 21 0 0 0 SL usbevt 0xffffff0003b4d420 [usb5] 20 0 0 0 SL usbevt 0xfffffffe403d7420 [usb4] 19 0 0 0 SL usbevt 0xfffffffe403d5420 [usb3] 18 0 0 0 SL usbevt 0xfffffffe403d3420 [usb2] 17 0 0 0 SL usbevt 0xfffffffe403d1420 [usb1] 16 0 0 0 SL usbtsk 0xffffffff80b53c28 [usbtask-dr] 15 0 0 0 SL usbtsk 0xffffffff80b53c00 [usbtask-hc] 14 0 0 0 SL usbevt 0xfffffffe403cf420 [usb0] 5 0 0 0 SL ccb_scan 0xffffffff80b214e0 [xpt_thrd] 13 0 0 0 SL - 0xffffffff80b58204 [yarrow] 4 0 0 0 SL - 0xffffffff80b54ba8 [g_down] 3 0 0 0 SL - 0xffffffff80b54ba0 [g_up] 2 0 0 0 SL - 0xffffffff80b54b90 [g_event] 12 0 0 0 RL (threaded) intr 100048 I [irq12: psm0] 100047 I [irq1: atkbd0] 100045 I [swi0: uart] 100042 I [irq14: ata0] 100040 L *Giant 0xffffff0004d2e5a0 [irq19: re0 ehci0] 100036 I [irq18: ohci2 ohci4] 100034 I [irq17: ohci1 ohci3] 100030 I [irq16: ohci0] 100029 I [irq22: atapci0] 100028 I [irq9: acpi0] 100026 RunQ [swi5: +] 100021 I [swi2: cambio] 100019 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 Run CPU 1 [swi4: clock] 100008 I [swi3: vm] 100007 I [swi1: net] 11 0 0 0 RL (threaded) idle 100006 CanRun [idle: cpu0] 100005 CanRun [idle: cpu1] 100004 CanRun [idle: cpu2] 100003 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xffffff00014d0890 [init] 10 0 0 0 SL audit_wo 0xffffffff80d287d0 [audit] 0 0 0 0 SLs (threaded) kernel 100043 D - 0xffffff0004cc7500 [fw0_taskq] 100027 D - 0xffffff000162f200 [thread taskq] 100025 D - 0xffffff000162f380 [kqueue taskq] 100024 D - 0xffffff000162f400 [acpi_task_2] 100023 D - 0xffffff000162f400 [acpi_task_1] 100022 D - 0xffffff000162f400 [acpi_task_0] 100016 D - 0xffffff00014e1400 [firmware taskq] 100000 D sched 0xffffffff80b54ca0 [swapper] 1108 1 1108 0 SE proctree 0xffffffff80b57ba0 shutdown 801 1 801 0 SE+ proctree 0xffffffff80b57ba0 getty db> allt Tracing command init pid 1111 tid 100117 td 0xffffff0004ec7a50 dev_refthread() at dev_refthread+0x33 cn_devopen() at cn_devopen+0x4e cnopen() at cnopen+0x42 giant_open() at giant_open+0x6b devfs_open() at devfs_open+0x184 VOP_OPEN_APV() at VOP_OPEN_APV+0xaf vn_open_cred() at vn_open_cred+0x308 kern_openat() at kern_openat+0x15e syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (5, FreeBSD ELF64, open), rip = 0x447d0c, rsp = 0x7fffffffe728, rbp = 0x7fffffffe7b0 --- Tracing command csh pid 819 tid 100114 td 0xffffff0014210a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2eb ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x84 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x54 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (3, FreeBSD ELF64, read), rip = 0x8009d6bbc, rsp = 0x7fffffffe4f8, rbp = 0x1 --- Tracing command su pid 818 tid 100082 td 0xffffff0004edc000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009d751c, rsp = 0x7fffffffe558, rbp = 0x333 --- Tracing command bash pid 811 tid 100106 td 0xffffff0004eab000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b8151c, rsp = 0x7fffffffe9b8, rbp = 0x6a7ce0 --- Tracing command sshd pid 810 tid 100063 td 0xffffff0004e25370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013b9b3c, rsp = 0x7fffffffdd38, rbp = 0x7fffffffddc0 --- Tracing command sshd pid 808 tid 100105 td 0xffffff0004eab370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe soreceive_generic() at soreceive_generic+0x1015 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x54 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013b9bbc, rsp = 0x7fffffffdd58, rbp = 0 --- Tracing command getty pid 807 tid 100116 td 0xffffff0004fb7000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x1a7 _sx_xlock() at _sx_xlock+0xc1 devfs_close() at devfs_close+0x4b VOP_CLOSE_APV() at VOP_CLOSE_APV+0xaf vn_close() at vn_close+0xb6 vn_closefile() at vn_closefile+0x80 devfs_close_f() at devfs_close_f+0x28 _fdrop() at _fdrop+0x23 closef() at closef+0x4d fdfree() at fdfree+0x178 exit1() at exit1+0x305 sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command getty pid 806 tid 100115 td 0xffffff0004fb7370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x1a7 _sx_xlock() at _sx_xlock+0xc1 devfs_close() at devfs_close+0x4b VOP_CLOSE_APV() at VOP_CLOSE_APV+0xaf vn_close() at vn_close+0xb6 vn_closefile() at vn_closefile+0x80 devfs_close_f() at devfs_close_f+0x28 _fdrop() at _fdrop+0x23 closef() at closef+0x4d fdfree() at fdfree+0x178 exit1() at exit1+0x305 sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command getty pid 805 tid 100067 td 0xffffff0004eaca50 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x3e5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804de4d9, rsp = 0xffffffff80d3a9d0, rbp = 0xfffffffeb6595500 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x109 _mtx_lock_flags() at _mtx_lock_flags+0xe1 count_dev() at count_dev+0x11 devfs_close() at devfs_close+0x210 VOP_CLOSE_APV() at VOP_CLOSE_APV+0xaf vn_close() at vn_close+0xb6 vn_closefile() at vn_closefile+0x80 devfs_close_f() at devfs_close_f+0x28 _fdrop() at _fdrop+0x23 closef() at closef+0x4d fdfree() at fdfree+0x178 exit1() at exit1+0x305 sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command getty pid 804 tid 100076 td 0xffffff0004e22000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2eb ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x84 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x54 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (3, FreeBSD ELF64, read), rip = 0x80083abbc, rsp = 0x7fffffffed68, rbp = 0 --- Tracing command getty pid 803 tid 100107 td 0xffffff0004e77a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x234 _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xe1 ttydev_close() at ttydev_close+0x26 devfs_close() at devfs_close+0x15c VOP_CLOSE_APV() at VOP_CLOSE_APV+0xaf vn_close() at vn_close+0xb6 vn_closefile() at vn_closefile+0x80 devfs_close_f() at devfs_close_f+0x28 _fdrop() at _fdrop+0x23 closef() at closef+0x4d fdfree() at fdfree+0x178 exit1() at exit1+0x305 sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command getty pid 802 tid 100077 td 0xffffff0004f42370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x234 _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xe1 ttydev_close() at ttydev_close+0x26 devfs_close() at devfs_close+0x15c VOP_CLOSE_APV() at VOP_CLOSE_APV+0xaf vn_close() at vn_close+0xb6 vn_closefile() at vn_closefile+0x80 devfs_close_f() at devfs_close_f+0x28 _fdrop() at _fdrop+0x23 closef() at closef+0x4d fdfree() at fdfree+0x178 exit1() at exit1+0x305 sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command getty pid 800 tid 100066 td 0xffffff0004e22a50 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x3e5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804de4d7, rsp = 0xfffffffe40017ff0, rbp = 0xfffffffeb65907b0 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x107 _mtx_lock_flags() at _mtx_lock_flags+0xe1 devfs_revoke() at devfs_revoke+0x91 VOP_REVOKE_APV() at VOP_REVOKE_APV+0xb5 exit1() at exit1+0x8cb sigexit() at sigexit+0x8f postsig() at postsig+0x32f ast() at ast+0x367 doreti_ast() at doreti_ast+0x1f Tracing command cron pid 765 tid 100074 td 0xffffff0004e226e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x235 kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800928fac, rsp = 0x7fffffffeba8, rbp = 0x3c --- Tracing command sendmail pid 759 tid 100112 td 0xffffff00141f6000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe kern_sigsuspend() at kern_sigsuspend+0xca sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d24f4c, rsp = 0x7fffffffce88, rbp = 0x1 --- Tracing command sendmail pid 755 tid 100092 td 0xffffff0004fd26e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x800db9b3c, rsp = 0x7fffffffc2f8, rbp = 0x7fffffffc390 --- Tracing command sshd pid 749 tid 100110 td 0xffffff0004e77000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013b9b3c, rsp = 0x7fffffffde48, rbp = 0x2 --- Tracing command ntpd pid 728 tid 100079 td 0xffffff0004edca50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x800d39b3c, rsp = 0x7fffffffec98, rbp = 0x7fffffffedc0 --- Tracing command nfsd pid 660 tid 100100 td 0xffffff0004efd6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe nfssvc() at nfssvc+0x327 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800695b2c, rsp = 0x7fffffffe9f8, rbp = 0x3 --- Tracing command nfsd pid 659 tid 100099 td 0xffffff0004efda50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe nfssvc() at nfssvc+0x327 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800695b2c, rsp = 0x7fffffffe9f8, rbp = 0x2 --- Tracing command nfsd pid 658 tid 100098 td 0xffffff0004f07000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe nfssvc() at nfssvc+0x327 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800695b2c, rsp = 0x7fffffffe9f8, rbp = 0x1 --- Tracing command nfsd pid 657 tid 100097 td 0xffffff0004f07370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe nfssvc() at nfssvc+0x327 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x800695b2c, rsp = 0x7fffffffe9f8, rbp = 0 --- Tracing command nfsd pid 656 tid 100064 td 0xffffff0004ead000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x800729b3c, rsp = 0x7fffffffea08, rbp = 0x7fffffffecd0 --- Tracing command mountd pid 654 tid 100080 td 0xffffff0004edc6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x80083eb3c, rsp = 0x7fffffffed08, rbp = 0x7fffffffee00 --- Tracing command rpcbind pid 603 tid 100084 td 0xffffff0004fb76e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2a4 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8008f46dc, rsp = 0x7fffffffcb68, rbp = 0x6100c0 --- Tracing command syslogd pid 587 tid 100078 td 0xffffff0004f42000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x80083db3c, rsp = 0x7fffffffdd58, rbp = 0x1 --- Tracing command devd pid 532 tid 100075 td 0xffffff0004e22370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x670 select() at select+0x56 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (93, FreeBSD ELF64, select), rip = 0x44034c, rsp = 0x7fffffffe8f8, rbp = 0x7fffffffe910 --- Tracing command softdepflush pid 27 tid 100056 td 0xffffff0004c4a370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 softdep_flush() at softdep_flush+0x2d0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb425bd30, rbp = 0 --- Tracing command syncer pid 26 tid 100055 td 0xffffff0004c4a6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4df fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4256d30, rbp = 0 --- Tracing command vnlru pid 25 tid 100054 td 0xffffff0004c4aa50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 vnlru_proc() at vnlru_proc+0x655 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4251d30, rbp = 0 --- Tracing command bufdaemon pid 24 tid 100053 td 0xffffff0004d2c000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 buf_daemon() at buf_daemon+0x208 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb424cd30, rbp = 0 --- Tracing command pagezero pid 23 tid 100052 td 0xffffff0004d2c370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4247d30, rbp = 0 --- Tracing command vmdaemon pid 22 tid 100051 td 0xffffff0004d2c6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4242d30, rbp = 0 --- Tracing command pagedaemon pid 9 tid 100050 td 0xffffff0004d2ca50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb423dd30, rbp = 0 --- Tracing command sctp_iterator pid 8 tid 100049 td 0xffffff0004d2d000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b sctp_iterator_thread() at sctp_iterator_thread+0x54 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4238d30, rbp = 0 --- Tracing command fdc0 pid 7 tid 100046 td 0xffffff0004d2da50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 fdc_thread() at fdc_thread+0x904 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4226d30, rbp = 0 --- Tracing command fw0_probe pid 6 tid 100044 td 0xffffff0004c48000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe fw_bus_probe_thread() at fw_bus_probe_thread+0xbc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4212d30, rbp = 0 --- Tracing command usb5 pid 21 tid 100041 td 0xffffff0004c48a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb412cd30, rbp = 0 --- Tracing command usb4 pid 20 tid 100039 td 0xffffff0004c49370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4110d30, rbp = 0 --- Tracing command usb3 pid 19 tid 100038 td 0xffffff0004c496e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4108d30, rbp = 0 --- Tracing command usb2 pid 18 tid 100037 td 0xffffff0004c49a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4100d30, rbp = 0 --- Tracing command usb1 pid 17 tid 100035 td 0xffffff000162e6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40f3d30, rbp = 0 --- Tracing command usbtask-dr pid 16 tid 100033 td 0xffffff0001632000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b usb_task_thread() at usb_task_thread+0x9c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40e6d30, rbp = 0 --- Tracing command usbtask-hc pid 15 tid 100032 td 0xffffff0001632370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b usb_task_thread() at usb_task_thread+0x9c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40e1d30, rbp = 0 --- Tracing command usb0 pid 14 tid 100031 td 0xffffff00016326e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 usb_event_thread() at usb_event_thread+0xab fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40dcd30, rbp = 0 --- Tracing command xpt_thrd pid 5 tid 100020 td 0xffffff000162e000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b xpt_scanner_thread() at xpt_scanner_thread+0x3a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4007bd30, rbp = 0 --- Tracing command yarrow pid 13 tid 100017 td 0xffffff00014e2000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 random_kthread() at random_kthread+0x194 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4006cd30, rbp = 0 --- Tracing command g_down pid 4 tid 100015 td 0xffffff00014e26e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 g_io_schedule_down() at g_io_schedule_down+0x250 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40062d30, rbp = 0 --- Tracing command g_up pid 3 tid 100014 td 0xffffff00014e2a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 g_io_schedule_up() at g_io_schedule_up+0x154 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4005dd30, rbp = 0 --- Tracing command g_event pid 2 tid 100013 td 0xffffff00014e3000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40058d30, rbp = 0 --- Tracing command intr pid 12 tid 100048 td 0xffffff0004d2d370 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100047 td 0xffffff0004d2d6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb422bd30, rbp = 0 --- Tracing command intr pid 12 tid 100045 td 0xffffff0001633a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4217d30, rbp = 0 --- Tracing command intr pid 12 tid 100042 td 0xffffff0004c486e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4139d30, rbp = 0 --- Tracing command intr pid 12 tid 100040 td 0xffffff0004c49000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x234 _mtx_lock_sleep() at _mtx_lock_sleep+0xe0 _mtx_lock_flags() at _mtx_lock_flags+0xe1 intr_event_execute_handlers() at intr_event_execute_handlers+0x5e ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb4115d30, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xffffff0004c4a000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40f8d30, rbp = 0 --- Tracing command intr pid 12 tid 100034 td 0xffffff000162ea50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb40ebd30, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xffffff0001632a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb30d5d30, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff0001633000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb2cb0d30, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xffffff0001633370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe400a3d30, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff00014e36e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xca ipi_bitmap_handler() at ipi_bitmap_handler+0x33 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x8f --- interrupt, rip = 0xffffffff804de172, rsp = 0xfffffffe40099970, rbp = 0xfffffffe400999a0 --- _mtx_unlock_flags() at _mtx_unlock_flags+0xc2 uma_zalloc_arg() at uma_zalloc_arg+0x654 re_newbuf() at re_newbuf+0x44 re_rxeof() at re_rxeof+0x1d7 re_int_task() at re_int_task+0x1f8 taskqueue_run() at taskqueue_run+0x96 intr_event_execute_handlers() at intr_event_execute_handlers+0x68 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40099d30, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xffffff000162da50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40080d30, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xffffff000162e370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40076d30, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xffffff00014dea50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40071d30, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00014e3370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40053d30, rbp = 0 --- Tracing command intr pid 12 tid 100011 td 0xffffff00014d4370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4004ed30, rbp = 0 --- Tracing command intr pid 12 tid 100010 td 0xffffff00014d46e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40049d30, rbp = 0 --- Tracing command intr pid 12 tid 100009 td 0xffffff00014d4a50 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x3e5 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff804de4ea, rsp = 0xfffffffe4000dff0, rbp = 0xfffffffe40044ae0 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x11a _mtx_lock_flags() at _mtx_lock_flags+0xe1 softclock() at softclock+0x23b intr_event_execute_handlers() at intr_event_execute_handlers+0x68 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40044d30, rbp = 0 --- Tracing command intr pid 12 tid 100008 td 0xffffff00014de000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xffffff00014de370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4003ad30, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xffffff00014de6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x33 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x8f --- interrupt, rip = 0xffffffff8079dfc6, rsp = 0xfffffffe40035b40, rbp = 0xfffffffe40035b50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a6 sched_idletd() at sched_idletd+0x223 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40035d30, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff00014d3000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x33 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x8f --- interrupt, rip = 0xffffffff804dec0b, rsp = 0xfffffffe40030b40, rbp = 0xfffffffe40030b90 --- _thread_lock_flags() at _thread_lock_flags+0xdb sched_idletd() at sched_idletd+0xf6 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40030d30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00014d3370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x33 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x8f --- interrupt, rip = 0xffffffff8079dfc6, rsp = 0xfffffffe4002bb40, rbp = 0xfffffffe4002bb50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1a6 sched_idletd() at sched_idletd+0x223 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4002bd30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff00014d36e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sched_idletd() at sched_idletd+0x24f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40026d30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00014d3a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x211 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x2fe kern_wait() at kern_wait+0x3cc wait4() at wait4+0x35 syscall() at syscall+0x1bf Xfast_syscall() at Xfast_syscall+0xc4 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c61c, rsp = 0x7fffffffe728, rbp = 0x457 --- Tracing command audit pid 10 tid 100001 td 0xffffff00014d4000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4001cd30, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xffffff0004c48370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x47 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffeb414dd30, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xffffff00016336e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4009ed30, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xffffff00014e3a50 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40094d30, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xffffff000162d000 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4008fd30, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xffffff000162d370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe4008ad30, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xffffff000162d6e0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40085d30, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00014e2370 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x34b taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffe40067d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80b550f0 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x335 scheduler() at scheduler+0x292 mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> call doadump Dumping 3967 MB (3 chunks) chunk 0: 1MB (156 pages) ... ok chunk 1: 3455MB (884432 pages) 3439 3423 3407 3391 3375 3359 3343 3327 3311 3295 3279 3263 3247 3231 3215 3199 3183 3167 3151 3135 3119 3103 3087 3071 3055 3039 3023 3007 2991 2975 2959 2943 2927 2911 2895 2879 2863 2847 2831 2815 2799 2783 2767 2751 2735 2719 2703 2687 2671 2655 2639 2623 2607 2591 2575 2559 2543 2527 2511 2495 2479 2463 2447 2431 2415 2399 2383 2367 2351 2335 2319 2303 2287 2271 2255 2239 2223 2207 2191 2175 2159 2143 2127 2111 2095 2079 2063 2047 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok chunk 2: 512MB (131072 pages) 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1 ... ok Dump complete = 0 db> reset